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ABSTRACT 


This  report  describes  the  activities  of  the  UCL  INDRA 
network  group  during  the  year  1978.  Amongst  the  subjects 
covered  are: 

Basic  implementation  of  the  X25  network  access  protocol, 
higher  level  protocol  above  X25,  network  interconnection; 
measurements  on  the  experimental  Satnet  network,  facsimile 
transmission  and  Simulation  of  network  performances. 

The  groups  using  the  UCL-ARPANET  link  during  1978  are 

listed,  and  the  INDRA  participation  on  work  on  national 

and  international  standards  for  data  communications  discussed. 
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I.  INTRODUCTION 

Each  year  the  Internetwork  Display  and  Remote  Access 
Group  (INDRA)  in  the  Department  of  Statistics  and  Computer 
Science,  University  College  London,  produces  an  annual 
report.  This  report  is  designed  to  summarise  our  work, 
and  is  a  contractual  commitment  to  many  of  the  organis¬ 
ations  supporting  our  work. 

As  in  the  previous  annual  report  (1),  this  year's  annual 
report  includes  not  only  material  written  specially, 
but  also  a  number  of  papers  published  elsewhere.  The 
papers  included  here  are  those  most  representative  of  our 
research;  survey  papers  or  bibliographies  are  excluded, 
unless  they  also  reflect  our  activities.* 

As  a  result  of  this  approach,  the  annual  report  contains 
very  different  levels  of  detail  in  the  various  subjects 
covered.  In  some  cases  only  a  summary  is  given  and  the 
references  are  cited;  in  others  detailed  papers  are 
presented.  Hence  the  length  of  chapters  should  not  be 
considered  as  indicative  of  the  relative  importance  of 
the  subject.  The  inclusion  of  a  paper  implies  only  that 
the  relevant  subject  has  produced  a  reasonably  finished 
piece  of  work  during  the  year. 

This  year  we  decided  that  our  PDP-9s  would  be  withdrawn 
during  1980.  As  a  result  we  have  considered  carefully 
what  equipment  we  would  require  next,  and  how  we  could 
ensure  a  continuity  of  internal  facilities  and  external 
services.  We  cbncluded  that  a  phased  changeover  during 
the  next  two  years  was  indicated  -  while  the  present 
generation  of  networks  remained  and  the  new  set  was 
becoming  operational.  Here  it  must  be  remembered  that 
at  least  a  year  is  needed  after  a  data  network  becomes 
available  before  a  significant  number  of  computers  have 
implemented  the  requisite  protocols  to  use  it.  This 
period  allows  extensive  interesting  investigation  of  the 
network's  properties,  but  not  real  service.  As  a  result 
of  these  considerations,  we  have  formed  a  plan  of  how  our 
system  should  develop  over  the  next  couple  of  years,  which 
is  briefly  summarised  in  Chapter  2. 

Two  themes  can  be  singled  out  clearly  as  being  important 
in  our  future  research.  One  is  a  concern  with  the  prop¬ 
erties  of  the  new  generation  of  X25  networks;  the  second 
is  the  consideration  of  the  problems  encountered  in  conn¬ 
ecting  networks  together.  A  number  of  UK  groups  are  con¬ 
cerned  with  the  first  topic,  comparatively  few  with  the 
second.  Several  US  groups  in  the  ARPA  community  are  con¬ 
cerned  with  the  second  problem,  few  with  the  first.  Thus 
we  are  in  a  fairly  unique  position.  We  will  shortly  have 
access  to  many  X25  networks  -  but  are  also  concerned  with 
their  interconnection  to  satellite,  local  ring,  and  packet 
radio  networks  in  addition  to  the  more  conventional 
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technologies.  Our  activities  on  these  two  themes  form 
the  core  of  our  present  work,  and  are  discussed  in 
Chapters  3  and  -1. 

For  several  years  we  have  been  involved  with  a  SATNET 
e.\periment.  Our  role  has  been  to  make  User  Level  measure¬ 
ments  of  that  network,  in  a  way  that  would  be  relevant  to 
other  technologies.  In  this  last  year  of  that  experiment, 
we  have  perfected  our  tools  and  made  many  valuable  measure¬ 
ments.  These  measurements  have  shown  up  performance 
deficiencies,  which  should  be  remedied  in  1979  as  SATNET 
becomes  an  operational  entity.  This  work  is  described  in 
Chapter  5. 

In  1978,  the  users  of  the  UK  Post  Office  (PO)  Experimental 
Packet  Switched  Service  (EPSS)  held  an  Open  Day,  during 
which  EPSS  was  demonstrated  at  5  major  sites;  one  of  these 
was  UCL.  Throughout  that  day  we  demonstrated  Network 
Concatenation,  Measurements  and  SATNET  altogether.  This 
activity  is  discussed  in  Section  4.5.  Network  interconnec¬ 
tions  are  readily  possible.  Thus  at  another  Open  Day  at 
the  European  Information  Network  (EIN),  both  EIN  and  our¬ 
selves  removed  our  customary  access  barriers.  Users  of 
EIN  Hosts  in  Switzerland,  Germany,  France  and  many  other 
countries  demonstrated  use  of  some  ARPANET  facilities 
through  a  concatenation  of  EIN,  EPSS  and  ARPANET.  It 
should  be  emphasised  that  such  use  is  not  possible  normally. 
The  relevant  authorities  on  both  sides  of  the  Atlantic  do 
not  permit  such  usage  and  technical  barriers  have  been 
imposed  to  make  such  usage  impossible  except  under  special 
approval . 

An  investigation  of  facsimile  services  over  Data  Networks 
has  been  a  constant  activity  of  the  group  for  some  years. 
During  this  year,  there  was  a  consolidation  phase.  Our 
first  project  terminated  in  the  middle  of  the  year.  Most 
of  the  effort  was  spent  in  finishing  off  the  old  project 
and  preparing  for  a  new  one  which  will  start  in  1979. 

Our  future  theme  will  involve  much  more  integration  of 
Office  Automation  functions  with  data  networks  and  computer 
facilities.  Our  work  in  1978  is  described  in  Chapter  6. 

We  started  an  exercise  several  years  ago  of  simulating 
specific  properties  of  network  protocols.  This  year,  for 
the  first  time,  two  publications  have  resulted  from  this 
work;  these  are  presented  in  Chapter  7.  Another  signifi¬ 
cant  INDRA  activity  is  the  support  of  'collaborative'  use 
of  ARPANET  facilities  between  US  and  UK  research  groups. 

The  activities  in  this  area  are  discussed  in  Chapter  8. 

The  widespread  use  of  data  networks  depends  on  easy  inter¬ 
connection  of  different  computer  systems  and  the  underlying 
data  networks.  For  this,  agreement  on  a  wide  range  of 
network  protocols  is  essential.  The  ensuing  standards 
activities  involve  many  bodies  with  UK  members  -  the 
International  Consultative  Committees  on  Telephone  and 
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Tt'lt'graph,  the  I nt ernut ionul  Standards  Organisation,  the 
British  Standards  Institute,  the  UK  Post  Office  Study 
Groups  and  various  EURONET  groups  are  Just  some  examples. 

The  Group  is  deeply  involved  with  several  of  these  bodies. 
Our  affiliations  are  listed  in  Chapter  9. 

The  key  constituent  of  a  research  group  is  its  mt'mbers. 

In  Chapter  10  the  names  of  the  group  members  are  listed 
individual ly . 

A  research  group  like  ours  must  be  Judged  largely  on  what 
it  produces.  The  products  can  be  excellent,  but  are  not 
useful  if  there  is  no  publication  of  the  results.  There¬ 
fore  our  1978  publications  are  listed  in  Chapter  11. 

This  report  selves  also  as  the  annual  one  to  a  number  of 
organisations  whose  support  is  gratefully  acknowledged. 

These  organisations  are  the  Atomic  Energy  Authority 
(Agreements  MIC  69324  and  AGMT/CU1./936 ) ,  the  British  Library 
(SI/G/172  and  221),  the  Ministry  of  Defence  (AT/2047/064),' 
the  Science  Research  Council  ( B/RCi/66168)  and  the  US 
Defence  Advanced  Research  Projects  Agency  via  the  Office 
of  Naval  Research  ( NOOO 14-77-6-0005 ) . 


II.  SYSTEM  DEVELOPMENT 


4  . 


2.1  Introduction 


The  INDRA  configuration  is  rapidly  changing  from  one  which, 
until  1977,  relied  heavily  on  a  couple  of  DEC  PDP-9s  and 
a  link  to  ARPANET.  Over  the  last  year  the  Gateway  to 
SATNET,  which  is  a  full-sized  PDPll/35  ( 128K  words)  has 
almost  become  operational.  A  similar  sized  machine  has 
been  obtained  for  software  development  and  general  use. 
Moreover,  a  philosophy  has  been  developed  by  which  dedi¬ 
cated  LSI-lls  will  be  used  for  specific  functions  of 
communication  switching  or  protocol  conversion.  With  this 
goal,  we  have  expanded  the  number  of  LSI-lls  to  three,  and 
expect  four  to  six  more  during  1979. 

Up  to  this  same  time  our  network  activities  have  been 
mainly  with  ARPANET  and  EPSS.  We  expect  to  give  up  our 
direct  ARPANET  link  in  mid-1979,  and  our  EPSS  one  early  in 
1980.  The  Satnet  link  is  becoming  operational  now.  We 
expect  to  obtain  links  to  Euronet,  IPSS  and  PSS  in  the 
second  half  of  of  1979.  A  link  to  an  RSRE  network  is 
expected  somewhat  earlier. 

In  view  of  these  developments  it  is  hardly  surprising  that 
a  major  activity  in  the  future  will  be  in  techniques  for 
network  interconnection  with  local  networks.  Some  of  the 
forward  thinking  on  this  subject  is  discussed  in  Chapter  3. 
Our  initial  activities  in  the  local  network  field  are 
mentioned  in  Section  2.4. 


2 . 2  The  Configuration  at  the  End  of  1978 

By  the  end  of  1978,  we  had  much  more  hardware  than  at  the 
beginning  of  the  year  -  but  little  more  was  operational 
from  a  networking  standpoint.  The  configuration  is  sket¬ 
ched  in  Fig.  2.1.  The  link  to  the  RSRE  GEC  4080  had  been 
removed,  because  the  proj-ect  of  evaluating  CORAL  66  from 
the  US  had  ended.  The  link  to  the  ULCC  CDC  6000/7000  had 
been  withdrawn;  this  was  partly  because  ULCC  needed  the 
line,  partly  because  the  CDC  6000/7000  software  was  not 
really  suited  to  this  type  of  networking  and  partly  because 
our  PDP-9S  were  overloaded.  Both  the  direct  links  of 
Culham  and  RL  to  ARPANET  were  permanently  available,  as 
were  both  the  links  to  EPSS.  These  were  taken  down  only 
for  system  development. 

The  Satnet  connection  was  via  a  PDP-11  Gateway,  and  was 
slowly  becoming  operational.  In  addition,  we  had  taken 
delivery  of  one  128K  word  PDP-11/34,  with  a  60M  byte 
disk,  and  a  total  of  3  LSI-lls.  The  PDP-11  (actually  a 
Systime  5000)  ran  UNIX.  It  supported  both  local  terminals, 
and  is  planned  to  have  asynchronous  links  to  the  LSI-lls. 
Software  for  down-line  loadine  over  the  asynchronous  links 
was  under  development. 


LSI-11 


Wo  hud  only  one  HDLC  intorfaco  for  the  LSI-lls,  but  the 
desitjn  for  PCHs  was  nearly  complete.  We  expect  to  be 
able  to  produce  several  10s  of  HDLC  interfaces  at  about 
i500  each  by  the  spring  of  1079. 

Hardware  existed  for  attaching  the  LSl-lls  and  the  SYSTIME 
to  the  TIP  -  but  lliere  was  no  port  available  on  the  TIP. 
One  of  the  LSI-lls  was  envisaj'ed  as  a  TIP  Port  Expander; 
it  would  allow  the  SYSTIME,  the  Gateway  and  up  to  3 
LSl-lls  to  be  attached  to  the  TIP.  Unfortunately  ihe 
softwart*  for  this  Port  Expander  PE  was  not  yet  available; 
it  was  coming  shortly  from  SRI. 


2 . 3  The  Planned  Configuration 

The  direction  we  are  going  is  illustrated  in  Eig.  2.2. 

The'  direct  connection  to  ARI'ANET  is  to  be  replaced  during 
1079,  by  total  reliance  on  the  Satnet  connection  for  transit 
to  the  US.  There  will  still  be  two  lines  to  the  SIMP. 

One  will  carry  the  normal  ARI’ANET  traffic  using  the  conven¬ 
tional  protocols;  the  second  will  go  through  the  Gateway 
and  use  the  newer  Transnet  ones.  The  present  PDP-11 
Satnet  Gateway  will  be  replaced  by  an  LSI -11  mini-Gateway , 
leaving  the  POP- 11  free  to  provide  Network  Services  such 
as  Mailbox,  Speech  and  Local  Filing.  A  new  Port  Expander 
will  allow  both  development  machine  and  services  1  such  as 
Facsimih')  to  be  attached  to  both  the  TIP  and  the  mini- 
Gateway.  Th<'  PDP-Os  will  continue  their  present  role 
until  EPSS  goes  and  PSS  is  operational.  A  new  LSl-11 
Terminal  Concentrator  will  be  introduced. 

Three  new  networks  will  come  into  the  scene  during  1979. 

These  are  the  UK  PO  Packet  Switched  Service  PSS,  the 
European  Euronet ,  and  the  Ministry  of  Defence  (Royal  Signal 
Research  Est.)  Pilot  Service  Packet  Switched  Network  (PSPSN). 
The  first  two  of  these  will  use  the  X25  Virtual  Call  Network 
Access.  No  direct  connection  is  shown  to  Euronet,  partly 
because  of  the  regulatory  implications  of  switching  inter¬ 
national  traffic,  and  partly  because  we  do  not  anticipate 
a  need  for  such  traffic.  In  principle  the  access  to 
Euronet,  IPSS  (the  US-UK  Packet  Switched  Service)  and  PSS 
are  identical  in  form,  though  in  practice  there  are  differ¬ 
ences  (particularly  in  level  2).  However,  if  we  can  do 
protocol  conversion  in  PC(VC)  for  one,  we  could  do  it  for 
the  other  easily  -  and  will  have  appropriate  access  lines. 
I'SPSN  also  will  use  at  least  level  2  X25  as  an  access  proto¬ 
col.  RSRPl  will  also  wish  to  experiment  with  a  datagram 
version  of  level  3  X25 ,  in  which  the  Transport  stations 
will  be  the  US  ARPA-developed  TCP;  in  addition,  they  expect 
to  experiment  also  with  the  virtual  call  form  of  connections. 
These  two  forms  of  connections  are  shown  diagrammat ical ly 
as  two  separate  protocol  convertors  for  virtual  call  and 
datagram  traffic.  The  Virtual  Call  Convertor  PC(VC)  must 
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ill  so  map  transport 
X3/X28/X29  and  th(* 
Terminal  Traffic; 
be  required  also, 
in  Chapter  4. 


station  protocols;  some  mapping  between 
ARPANET  Telnet  will  be  required  for  the 
a  version  of  bulk  transfer  service  will 
These  subjects  are  mentioned  further 


Almost  all  the  LSl-lls  are  shown  with  asynchronous  lines 
to  the  UCL  Development  Machine  (DM)  running  UNIX.  These 
lines  will  exist  to  all  the  LSIs,  and  permits  the  boot¬ 
strapping  of  software  off  the  DM  disk.  A  "Black  Box"  from 
Sitintel,  suitable  for  connecting  Hosts  to  Euronet  by  pseudo 
"Character  Terminal"  connections,  is  being  obtained  as  part 
of  the  Euronet  project.  It  is  not  yet  clear  whether  or  how 
this  will  integrate  with  the  other  facilities.  Finally, 
we  expect  to  offer  an  experimental  facsimile  service  over 
both  Satnet-ARI’ANET  and  PSS ;  this  is  the  reason  for  the 
LSI- 11  shown  as  FAX. 


2.4  Local  Network  Plans 


The  necessary  connections  of  PDP-lls,  LSI-lls  and  other 
machines  shown  in  Fig.  2.2  are  the  direct  result  of  the 
types  of  hardware  and  software  facilities  now  available. 
Clearly  the  configuration  has  no  long-te*.n  generalisable 
structure,  and  will  not,  one  hopes,  ever  be  replicated. 

A  clear  requirement  is  for  some  kind  of  local  network  conn¬ 
ecting  the  INDRA  resources  with  Gateways  to  the  Wide  Area 
Networks.  Our  planned  developments  are  outlined  in  Fig.  2.3. 

There  are  already  may  such  local  networks  under  development, 
and  the  resources  of  our  Group  are  stretched.  For  this 
reason  we  intend  to  use  a  copy  of  the  Cambridge  Ring  System 
(Fig.  2.3).  To  allow  easy  replication,  we  are  arranging 
the  designs  of  the  relevant  boards  in  printed  circuit  board 
technology.  The  ring  will  include  not  only  INDRA  computers, 
but  also  others  in  the  department  (and  eventually  perhaps 
other  departments).  The  connections  may  include  not  only 
Satnet,  PSS  and  EPSS ,  but  also  other  College  and  University 
computing  resources. 

At  present  we  are  aiming  at  a  conventional  10Mbps  transfer 
rate.  The  ring  transfers  two  bytes  in  a  packet,  so  that 
the  devices  attached  may  use  any  mutually  agreed  protocol 
structure  above  the  basic  packet  transport  level.  We  have 
not  really  started  specifying  the  necessary  local  pro¬ 
tocols  or  their  transformations  in  MG  and  PC  for  Wide  Area 
Networks . 

The  ring  has  three  components;  a  Repeater,  a  Station  and  a 
Monitor  Station.  By  the  end  of  1978,  the  Repeater  design 
had  gone  out  for  PCB  prototyping.  The  Station  is  expected 
to  be  prototyped  also  during  the  first  quarter  of  1979. 

A  complete  3-station  prototype  system  (but  with  the  compo¬ 
nents  already  on  PCBs)  should  be  ready  by  the  end  of  the 
summer  of  1979.  In  addition,  each  station  needs  a  host- 
dependent  interrupt.  Programme  interrupt  versions  for  the 
PDPll  have  been  developed  by  Cambridge  University;  we  will 
design  a  DMA  version. 


ventual  Systems 
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III.  xuf)  HKLATKI)  ACTIVITIKS 
.'$.1  Introduction 


In  view  of  the  widespread  European  and  UK  interest  in  tiio 
PTT  X-series  Recommendations,  we  have  a  substantial  acti¬ 
vity  in  this  area.  The  activity  includes  hardware  (Section 
3.2),  basic  software  for  Network  Access  (Section  3.3)  and 
Protocol  Studies  (Section  3.4).  In  addition  much  of  our 
network  interconnection  work  is  related  to  these  specifica- 
t ions . 


3.2  Hardware 


Uy  the  end  of  the  year  we  had  such  confidence  in  our 

LSI -11  HULC  hardware,  that  we  had  completed  drawings  for 
PCS  manufacture.  This  particular  version  has  one  Program 
Interrupt  HDLC  interface  per  board,  and  will  cost  only  a 
fi'w  hundretl  pounds  to  replicate.  Because  it  will  control 
also  tlu'  facsimile  device  (see  Chapter  5),  we  expect  to 
need  several  tens  of  these  interfaces.  Because  they  are 
Program  Interrupt  per  character,  and  all  levels  2  and  3 
ar('  done  by  software  in  the  processor,  the  speed  of  the 
interface  will  be  limited  to  about  5  Kbps.  Various  alter¬ 
natives  are  being  studied  for  a  higher  speed  version; 
at  least  one  such  will  be  built. 

A  Pl)P-‘)  v«'rsion  of  the  HDLC  interface  was  also  developed. 
Here  we  modified  a  previous  DMA  device,  and  produce  inter¬ 
rupts  only  at  the  end  of  the  packet. 


3.3  Basic  software 


Implementations  of  the  INTEL  8080  and  PDP-9  versions  of 
X2.'i  were  largely  completed.  The  INTEL  8080  version  proved 
too  big  to  coexist  with  the  facsimile  software  in  one  8080. 
Our  hardware  does  not  allow  multiprocessor  working,  and  as 
a  group  VO  are  developing  support  facilities  mainly  for 
tlie  LSI-lls  at  present.  For  these  reasons  the  INTEL  8080 
development  has  been  abandoned. 

The  PDP-9  development  is  very  slow,  because  the  person 
working  on  it  has  left.  We  hope  to  complete  this  develop¬ 
ment  in  1979. 


The  PDP-11  development  has  made  good  progress.  Here  our 
early  work  was  in  HTL/2  under  RSX  11.  We  will  want  to 
transfer  the  software  to  run  under  UNIX  and  MOS  also.  In 
ordt'r  to  reduce  its  size,  and  increase  its  efficiency,  the 
software  has  been  reprogrammed  in  Assembler,  we  have  arran¬ 
ged  also  that  it  interfaces  to  the  operating  systems  at 
only  one  module  (COMSYS).  These  two  factors  will  make  it 
considerably  easier  to  convert  the  software  to  run  under 
UNIX  and  MOS,  both  of  which  we  need  for  the  development 
of  Fig.  2.2.  Clearly  interfaces  to  other  operating  systems 
could  be  developed  also.  The  size  of  both  the  RTL/2  and 
Assembly  versions  are  shown  in  Table  3.1. 
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RTL/2 

- 1 

Assembler 

Frame  Level  (level  1) 

— 

i 

Link  Level  (level  2) 

4 

1 

Packet  Level  (level  3) 

8 

2i 

COMSYS 

- 

J 

OS  Interface 

- 

i 

TOTAL 

12 

5 

Table  3.1  Size  of  Different  Modules  in  Kwords 


The  level  2  has  been  tested  successfully  with  the  Post 
Office  tester.  The  level  3  will  be  tested  early  in  1979. 
An  overview  has  been  written  (4);  both  the  level  2  and  the 
Framt'  level  of  the  software  have  been  fully  documented 
(5, 6, 7, 8).  It  can  be  extended  easily  to  handle  multiple 
transmission  lines  (9). 


4  X25  and  Hif;h  Level  Protocol  Studies 

We  have  made  studies  of  several  significant  problems  in 
the  protocol.  The  most  important  of  these  studies*  how 
to  avoid  asymmetries,  is  presented  below.  In  addition,  we 
have  writtem  one  tutorial  paper  (10),  and  commented  on 
other  X25  studies  (11).  We  have  specified  a  "Black  Box" 
for  connecting  Hosts  to  X25  Networks  using  only  their 
character  terminal  communications  interfaces  (12). 

Finally,  we  have  made  a  substantial  study  of  Remote  Printing 
Protocols  for  Euronet  (13);  this  document  is  being  published 
by  the  Commission  of  the  European  Economic  Community. 


3.5  X25  Asymmetries  and  How  to  Avoid  Them 
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INTRODUCTION 

The  X25  specification  defines  the  way  of  connecting  Data 
Terminal  Equipment  (DTE)  to  Data  Circuit-terminating  Equipment 
(DCE).  This  standard  has  differences  between  the  way  a  DTE 
functions  and  the  way  a  DCE  functions,  thus  giving  an 
asymmetrical  interface.  In  this  paper  we  show  that  it  is 
possible  to  make  a  set  of  choices  so  that  the  computer 
software  will  operate  in  either  DTE  or  DCE  mode.  Moreover,  we 
show  how  the  software  can  make  decisions  so  that  it  will 
switch  itself  into  the  appropriate  mode  while  remainin'^ 
completely  within  the  formal  specification.  We  regard  such 
software  as  being  highly  desirable  for  use  in  situations  where 
a  host  on  one  network  is  a  switching  node  on  another  network. 

The  X25  specification  is  divided  into  a  number  of  sections. 
The  original  Link  Access  Procedure  (LAP)  has  been  replaced  by 
LAPB.  Although  new  implementations  will  use  LAPB,  older 
implementations  are  already  using  LAP,  so  we  may  expect  that 
there  will  be  a  transition  period  in  which  implementations 
will  be  required  which  are  capable  of  switching  between  LAP 
and  LAPB.  We  show  how  this  may  be  accomplished  while 
remaining  completely  within  the  formal  specification. 

In  the  penultimate  section  of  this  paper,  we  propose  a  minor 
addition  to  the  formal  specification  in  order  to  allow  error 
notification  and  recovery  in  a  consistent  manner.  Whilst  it 
is  recognised  that  it  is  difficult  to  make  changes  to  the 
formal  specification  at  this  late  stag«,  it  is  also  recognised 
that  there  will  have  to  be  some  modif icatians  in  order  to  take 
care  of  existing  problems  [1].  We  hope  that  the  contents  of 
this  paper  will  enable  a  complete  set  of  problems  to  be 
identified  rather  than  having  the  specification  updated  in  a 
piecemeal  fashion  as  each  new  problem  is  encountered. 
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BACKGROUND 


The  INDRA  research  group  at  UCL  Is  involved  in  work  on  several 
computer  networks,  principly  Arpanet  [2],  Satnet  [3]f  EPSS 
(The  Experimental  Packet  Switched  Service  of  the  UK  Post 
Office)  [4],  and  Euronet  [5].  Our  activities  include 
Investigating  and  implementing  gateway  software  to  connect 
these  networks  together;  for  example,  the  gateway  between 
Arpanet  and  EPSS  is  available  as  a  service  to  users  of  these 
networks.  It  is  clear  that  the  UK  successor  to  EPSS,  which  we 
call  PSS,  will  be  an  X25  network  in  common  with  the  national 
networks  of  other  countries  involved  in  Euronet.  In  view  of 
this  adoption  of  the  X25  standard,  we  decided  that,  in  at 
least  one  mode,  our  gateways  should  make  each  network  appear 
as  an  X25  network  to  any  other  machine  connected  to  the 
gateways. 

The  requirements  of  the  gateway  X25  implementations  are  that 
they  communicate  with  DTE  implementations  (for  applications 
such  as  facsimile  terminals  [6])  and  with  each  other  (for 
inter-network  traffic);  the  local  PSS  exchange  will 
undoubtedly  expect  our  implementations  to  appear  as  DTEs. 
From  these  considerations,  it  became  apparent  that  a  gateway 
X25  implementation  must  take  on  the  appearance  expected  of  it 
by  the  remote  station  (DTE  or  DCE).  Remote  stations  fall  into 
three  categories:  strict  DCEs  which  expect  this  station  to 
adhere  to  the  (implied)  specification  for  a  DTE,  strict  DTEs 
which  expect  this  station  to  adhere  to  the  formal 
specification  for  a  DCE,  and  non-strict  stations  which  allow 
this  station  to  use  either  the  DTE  or  the  DCE  form  of  the 
protocol.  The  non-strict  category  is  subdivided  into  adaptive 
and  non-adaptlve  stations;  adaptive  stations  use  information 
derived  from  the  incoming  data  stream  to  modify  their 
appearance,  whereas  non-adaptive  stations  always  appear  the 
same.  The  gateway  X25  implementations  are  necessarily 
adaptive. 

PROPOSED  STRATEGY 

The  underlying  strategy  used  in  adaptive  stations  is  first  to 
assume  that  the  remote  station  is  a  DCE  and  then  modify  this 
assumption  when  the  remote  station  indicates  that  an  error  has 
been  made;  the  sections  below  describe  the  modifications  in 
detail.  This  strategy  is  followed  Independently  at  level  2 
and  level  3  of  X25  since  either  of  these  may  be  used  with 
other  protocols. 
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As  an  added  complication,  the  X2b  LAP  specification  has  been 
replaced  by  the  LAPB  specification.  Implementations  started 
before  this  change  are  based  on  LAP,  while  those  started  later 
are  based  on  LAPB.  It  is  not  yet  clear  whether  the  UK  PSS 
will  provide  LAP  access,  LAPB  access,  or  both.  Presented 
below  are  studies  of  the  asymmetries  in  networks  which  use  LAP 
only,  networks  which  use  LAPB  only,  and  networks  which  use 
both  LAP  and  LAPB.  In  these  studies,  we  refer  extensively  to 
the  published  X25  specification  [7,8,9],  and  the  section 
references  In  brackets  {}  refer  to  sections  in  these 
documents . 

LAP-ONLY  NETWORKS 

The  only  asymmetry  in  the  LAP  specification  is  in  the  content 
of  the  frame  address  field:  DCEs  use  address  A  for  commands 
and  address  B  for  responses,  while  DTEs  use  address  B  for 
commands  and  address  A  for  responses  [Section  2.^.2]. 

Since  the  type  of  any  particular  frame  is  uniquely  determined 
by  the  frame  command  byte,  it  is  possible  to'ignore  the 
address  bytes  of  incoming  frames  (with  the  possible  exceptio.n 
of  validating  that  the  address  is  either  A  or  B) .  It  has  been 
suggested  that  this  will  result  in  an  implementation  which  is 
less  robust  (more  prone  to  failure)  than  one  which  uses  the 
address  field  for  the  detection  of  looped  communications 
lines.  We  feel  that  algorithms  for  the  detection  of  looped 
lines  are  error  prone  in  themselves;  for  example,  the  dropping 
of  the  low  order  bit  of  the  address  field  may  result  in  a 
station  deciding  that  the  line  has  become  looped.  We  observe 
that  a  strict  DCE  transmitting  any  response  frame  down  a 
looped  line  will  consequently  transmit  CMDR  frames  ad 
infinitum  (Section  2 .8 .2) .  We  also  note  that  a  line 
becoming  looped  (but  undetected)  may  affect  the  higher  level 
protocols;  for  example,  a  level  3  Call  Request  will  always 
collide  with  itself. 

For  outgoing  frames,  this  station  should  first  assume  DTE 
address  conventions  (B  for  commands  and  A  for  responses).  If 
the  remote  station  is  either  a  strict  DCE,  expecting  DTE 
address  conventions,  or  a  non-strict  station,  accepting  either 
DTE  or  DCE  address  conventions,  this  assumption  is  correct. 
If  the  remote  station  is  a  strict  DTE,  expecting  DCE  address 
conventions,  either  the  outgoing  SARM  or  the  UA  sent  in  reply 
to  the  incoming  SARM  will  cause  a  CMDR  response  from  the 
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remote  station  indicating  invalid  command  {Section  2. <4. 8. 2}. 
On  receipt  of  this  CMDR  frame,  this  station  should  switch  to 
DCE  address  conventions  and  transmit  another  SARM.  The  remote 
station  will  retransmit  its  SARM  after  a  timeout. 

Bit  13  of  the  information  field  of  a  CMDR  frame  indicates 
whether  the  frame  being  rejected  was  a  command  or  a  response 
frame  {Section  2.3.^.  10).  The  polarity  of  this  bit  in  an 
outgoing  CMDR  t>ame  can  be  determined  by  saving  the  address 
byte  of  the  incoming  SARM  for  comparison  with  the  address  byte 
of  erroneous  frames. 

LAPB-ONLY  NETWORKS 

The  asymmetries  in  the  LAPB  specification  are  as  follows: 

-  DCEs  use  address  A  for  commands  and  address  B  for 
responses,  while  DTEs  use  address  B  for  commands  and 
address  A  for  responses  {Section  2.4.2). 

-  SABM  is  transmitted  by  DTEs  only,  although  the  use  of 
SABM  by  DCEs  is  for  further  study  {Section  2. 4. 5.1). 

-  When  the  link  first  comes  up,  DCEs  may  transmit  DM  frames 
{Sections  2. 3. 4. 9  and  2. 4. 5. 4. 2). 

-  When  receiving  a  CMDR/FRMR  frame,  DCEs  transmit  a  DM 

response  {Section  2.4.10.2  -  the  phrase  "it  is  a  UA  or  DM 
response”  should  read  "it  is  a  CMDR/FRMR  or  DM 

response" ) . 

-  DCEs  transmit  RR,  RNR,  and  REJ  response  frames,  but  do 
not  transmit  RR,  RNR,  or  REJ  command  frames  {Section 

•2.3.4).  This  asymmetry  is  avoided  by  never  transmitting 
RR,  RNR,  or  REJ  command  frames  but  always  responding  to 
such  frames. 

As  with  LAP,  this  station  should  first  assume  DTE  conventions 
and  transmit  a  SABM  frame  containing  address  B;  incoming  DM 
frames  should  be  Ignored.  If  the  remote  station  is  a  strict 
DCE  or  a  non-strict  station,  the  assumption  is  correct  and  a 
UA  response  will  be  returned;  any  incoming  SABM  frame  should 
be  answered  with  a  UA  response.  If  the  remote  station  is  a 
strict  DTE,  the  outgoing  SABM  Or  the  UA  transmitted  in 
response  to  the  incoming  SABM  may  cause  the  remote  station  to 
transmit  a  CMDR/FRMR  response;  on  receipt  of  this  frame,  this 
station  should  switch  to  DCE  conventions.  Alternatively,  the 
remote  station  may  Ignore  both  the  SABM  and  UA  frames  and, 
after  the  timeout  period,  retransmit  its  own  SABM;  if  this 
persists  for  some  time  (such  as  half  of  N2  timeout  periods), 
this  station  should  then  switch  to  DCE  conventions. 
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In  reply  to  a  CMDR/FRMR  frame,  this  station  should  transmit  a 
DM  response.  A  strict  DTE  or  non-strict  station  will  reply  to 
this  with  a  SABM  command;  a  strict  DCE  will  reply  with  another 
DM  response,  to  which  this  station  should  reply  with  a  SABM 
command . 

MIXED  LAP  AND  LAPB  NETWORKS 

The  initial  problem  here  is  that  SARM  and  DM  are 
indistinguishable  until  the  address  conventions  of  the  remote 
station  are  known.  As'with  LAPB,  this  station  should  first 
assume  LAPB  DTE  conventions  and  transmit  a  SABM  frame 
containing  address  B.  If  the  remote  station  is  a  strict  LAPB 
DCE  or  a  non-strict  LAPB  station,  the  assumption  is  correct 
and  a  UA  response  will  be  returned  (as  in  the  LAPB  case).  If 
the  remote  station  is  a  strict  LAPB  DTE,  there  will  be  an 
incoming  SABM  frame  and  communication  can  be  established  as  in 
the  LAPB  case.  If  the  remote  station  is  a  strict  LAP  DCE  or  a 
non-strict  LAP  station,  the  outgoing  SABM  frame  will  cause  the 
remote  station  to  transmit  a  CMDR/FRMR  response;  the  address 
and  information  fields  of  this  response  uniquely  determines 
what  the  remote  station  is.  If  the  remote  station  is  a  strict^ 
LAP  DTE,  it  will  either  respond  to  the  SABM  frame  with  a  FRMR 
response,  uniquely  identifying  itself,  or  it  will  continually 
ignore  SABM  frames  (as  invalid  responses  to  its  SARM  command) 
and  retransmit  SARM  commands  whenever  the  timer  runs  out.  If 
this  latter  situation  persists  for  some  time  (such  as  half  of 
N2  timeout  periods),  this  station  should  switch  to  using  DCE 
address  conventions  and  transmit  another  SABM  frame;  this  will 
cause  the  remote  station  to  transmit  a  CMDR  response  as  above. 
Transmitting  a  SABM  frame  rather  than  a  SARM  frame  enables 
this  station  to  adopt  a  common  strategy  for  strict  LAP  DTEs 
and  strict  LAPB  DTEs. 

WINDOWING 

The  X25  specification  states  "The  value  of  the  Send  State 
Variable  is  incremented  by  one  with  each  successive 
Information  frame  transmission,  but  cannot  exceed  N(R)  of  the 
last  received  frame  by  more  than  the  maximum  number  of 
outstanding  frames  (k)."  {Section  2.3.2.^.!}  where  the  value 
of  k  is  agreed  between  the  DTE  and  DCE  administrations.  In 
the  absence  of  such  an  agreement,  the  consequences  of  choosing 
any  particular  value  for  k  must  be  investigated. 
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The  X25  specification  originally  stated  that  the  receipt  of  an 
invalid  N(S)  would  cause  a  CMDR  response  to  be  transmitted 
{Section  2.H.6.2}.  The  revisions  to  the  specification  have 
removed  an  invalid  N(S)  from  the  list  of  conditions  for 
transmitting  a  CMDR  response  (Section  2.‘i.8.2].  We  Interpret 
this  to  mean  that  any  I-frame  containing  an  invalid  N(S) 
should  be  discarded  (as  though  the  receiver  was  in  the  busy 
condition).  By  using  this  interpretation  of  the  revised 
specification,  the  value  of  k  for  receiving  can  be  set  to  any 
value  up  to  the  permitted  maximum.  For  transmitting,  the 
value  of  k  should  initially  be  set  to  the  permitted  maximum. 
If  the  remote  station  subsequently  regards  some  N(S)  as 
invalid  and  responds  with  a  CMDR  frame  (recognisable  by  having 
zero  in  the  third  information  field  byte),  this  station  should 
reduce  the  transmit  value  of  k  by  one.  Repeated  application 
of  this  algorithm  results  in  the  transmit  value  of  k  being 
reduced  to  match  the  receive  value  of  k  at  the  remote  station. 

Windowing  at  level  3  is  a  similar  mechanism  to  that  at 
level  2,  but  is  slightly  more  complex  in  that  an  N(S)  error 
will  involve  a  resetting  of  the  virtual  call  and  the 
possibility  of  consequential  data  loss.  The  only  safe 
solution  is  to  set  the  transmit  value  of  k  to  one  and  the 
receive  value  of  k  to  the  maximum  permitted  value. 

LEVEL  3  ASYMMETRIES 

The  asymmetries  in  the  level  3  specification  are  as  follows: 

-  When  selecting  a  logical  channel  for  an  outgoing  call, 
'the  DCE  uses  the  lowest  numbered  logical  channel  which  is 

in  the  ready  state,  while  the  DTE  uses  the  highest 
numbered  logical  channel  which  is  in  the  ready  state 
(Sections  3.'l«2  and  3*1«3}. 

-  When  a  call  collision  occurs,  the  DCE  clears  the  Incoming 
Call  and  continues  with  the  Call  Request;  the  DTE  ignores 
the  Incoming  Call  packet  (Section  3.1*6).  (Note  that  in 
X7x  both  calls  are  cleared). 

-  The  DCE  does  not  use  the  level  3  REJ  command  (Section 
4.1.4);  in  some  networks,  DTEs  also  may  not  use  the  level 
3  REJ  command.  This  asymmetry  is  avoided  by  not  using  or 
providing  this  facility. 

One  of  the  problems  in  connecting  to  a  system  of  unknown 
attributes  is  that  the  number  of  the  "highest"  logical  channel 
available  is  also  unknown.  To  resolve  this  problem,  it  is 


necessary  to  define  the  action  taken  by  any  station  when  it 
receives  a  packet  on  a  logical  channel  which  doesn't  exist 
(froa  the  stations  point  of  view).  We  propose  that  this 
action  is  specified  as  follows: 

"If  the  incoMing  packet  is  a  Clear  Confirmation 
packet,  ignore  it.  If  the  incoming  packet  is  a 
Clear  packet  (Clear  Request  or  Clear  Indication), 
send  out  a  Clear  Confiraation  packet  using  the  same 
logical  channel  number  as  used  in  the  incoming 
packet.  If  the  incoming  packet  is  not  a  Clear  or 
Clear  Confirmation  packet,  send  out  a  Clear  packet 
using  the  same  logical  channel  number  as  used  in  the 
incoming  packet  and  with  a  qualifier  indicating  that 
the  logical  channel  number  is  invalid." 

For  the  general  case,  we  partition  the  logical  channels  into 
two  sets  which  we  term  "high"  and  "low";  the  low  set  initially 
contains  only  the  two  lowest  numbered  logical  channels,  while 
the  remaining  logical  channels  are  in  the  high  set.  If 
something  odd  happens  on  one  of  the  low  channels  (such  as  a 
Call  Accepted  packet  being  received  without  a  Call  packet 
having  been  sent  out),  the  low  set  is  extended  to  include  the 
lowest  numbered  logical  channel  from  the  high  set  which  is  in 
the  ready  state  (and  on  which  nothing  odd  has  happened).  The 
low  set  thus  contains  at  least  two  logical  channels  in  the 
ready  state  at  both  stations  (if  this  is  possible).  In  the 
following  discussion,  we  denote  by  "channel  1"  the  lowest 
numbered  logical  channel  from  the  low  set  which  is  in  the 
ready  state,  and  by  "channel  2"  the  highest  numbered  logical 
channel  from  the  low  set  which  is  in  the  ready  state.  The 
strategy  requires  that  channel  1  and  channel  2  are  distinct 
while  there  are  two  or  more  logical  channels  in  the  ready 
state.  (The  low  set  may  be  defined  as  being  all  logical 
channels  with  numbers  less  than  or  equal  to  that  of  channel 
2). 

LEVEL  3  PROCEDURES 

The  initial  strategy  is  the  same  as  that  used  at  level  2: 
assume  DTE  conventions  until  it  is  known  what  the  remote 
station  is.  This  station  exists  in  one  of  four  modes: 
provisional  DTE  (the  initial  mode),  provisional  DCE,  confirmed 
DTE,  and  confirmed  DCE;  mode  changes  which  can  occur  are  from 
either  provisional  mode  to  any  other  mode  (these  changes  are 
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described  below).  The  first  call  to  be  made  gives  us  three 
cases  to  consider:  an  outgoing  call,  an  incoming  call,  or  a 
call  collision. 

An  incoming  call  will  be  on  either  channel  1  or  some  other 
logical  channel  in  the  ready  state  (unless  there  is  only  one 
free  logical  channel  left  due  to  outgoing  calls);  this 
determines  what  the  remote  station  is.  This  station  should 
switch  to  confirmed  DCE  mode  or  confirmed  DTE  mode  as 
appropriate.  If  the  incoming  call  uses  the  only  remaining 
free  logical  channel,  nothing  can  be  determined  about  the 
remote  station  so  no  mode  change  occurs  and  the  next  call  to 
be  made  is  again  regarded  as  the  first  call. 

An  outgoing  call  should  use  the  lowest  numbered  logical 
channel  from  the  high  set  which  is  in  the  ready  state.  If  the 
logical  channel  number  is  valid,  this  call  will  be  acceptable 
regardless  of  what  the  remote  station  is  (since  the  remote 
station  cannot  determine  the  state  of  the  logical  channels  as 
seen  by  this  station).  If  the  call  is  cleared  with  an 
"invalid  channel"  qualifier,  this  channel  and  all  others  with 
higher  numbers  should  be  removed  from  the  high  set  and  marked 
"not  to  be  used";  the  call  can  then  be  reattempted  on  another 
logical  channel.  If  there  are  no  logical  channels  other  than 
channel  1  and  channel  2  in  the  ready  state,  the  outgoing  call 
should  use  channel  2  if  this  Is  in  the  ready  state  and 
channel  1  otherwise  (if  possible).  No  mode  changes  occur  at 
this  station  because  of  this  call,  and  the  next  call  to  be 
made  (by  either  station)  is  again  regarded  as  the  first  call. 

If  a  call  collision  occurs  on  the  only  available  logical 
channel,  this  station  should  clear  its  outgoing  call  and 
continue  with  the  incoming  call;  this  action  is  necessary 
since  the  remote  station  may  be  a  strict  DTE.  (An  alternative 
is  to  clear  both  calls  as  in  X7x).  Note  that  if  both  stations 
take  this  DCE  action,  the  channel  will  be  cleared  when  either 
station  transmits  a  Call  Accepted  packet.  If  this  station 
receives  a  Call  Accepted  packet  before  transmitting  its  Call 
Accepted  packet,  it  should  clear  both  calls  and  change  to 
confirmed  DTE  mode  (since  the  remote  station  has  DCE 
potential).  If  the  outgoing  Call  Accepted  packet  does  not 
result  in  the  call  being  cleared  with  a  "procedure  error" 
qualifier,  the  remote  station  is  a  DTE  so  this  station  should 
change  to  confirmed  DCE  mode.  Some  implementations  may  be 
able  to  defer  clearing  the  outgoing  call  until  the  incoming 
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call  is  either  accepted  or  refused;  for  two  adaptive  stations, 
this  sets  up  a  race  condition  on  the  Call  Accepted  packets 
which  could  resolve  the  conflict,  particularly  when  one  of  the 
stations  is  an  inter-network  gateway. 

If  a  call  collision  occurs  on  a  logical  channel  other  than 
channel  1  and  outgoing  calls  have  previously  been  attempted, 
the  remote  station  is  known  to  be  a  strict  DTE  so  this  station 
should  immediately  switch  to  confirmed  DCE  mode  and  act 
accordingly . 

The  only  case  left  to  consider  is  that  of  a  call  collision 
when  no  previous  calls  have  been  made.  As  in  the  case  of  a 
call  collision  on  the  only  available  logical  channel,  this 
station  should  clear  its  outgoing  call  and  continue  with  the 
incoming  call.  Here,  however,  this  station  can  base  its 
future  strategy  on  the  contents  of  the  call  packets  by  doing  a 
byte-by-byte  comparison  and  using  the  first  pair  of  dissimilar 
bytes  to  make  a  decision:  if  the  byte  from  the  incoming  packet 
has  higher  value  (byte  values  are  in  the  range  0  to  255) 
switch  to  provisional  DCE  mode  and  use  channel  1  for.  the  next 
outgoing  call,  whereas  if  the  byte  from  the  outgoing  packet 
has  higher  value  switch  to  provisional  DTE  mode.  If  the 
packets  are  identical,  this  station  should  remain  in  its 
current  mode  until  more  significant  information  arrives. 
Comparing  the  whole  packets  is  rather  better  than  comparing, 
for  example,  the  DTE  address  fields  since  the  contents  of  the 
latter  usually  depend  on  the  X25  implementations  whereas  the 
contents  of  the  Call  User  Data  Field  depends  on  the 
applications  making  the  calls.  Note  that  an  arbitrary 
decision  has  been  made  on  the  direction  of  the  test;  the 
remote  station  may  use  different  criteria,  resulting  in  the 
race  condition  not  being  cleared  despite  the  fact  that  this 
station  was  able  to  make  a  decision. 

CONCLUSIONS 

This  study  shows  that  there  is  a  "universal”  X25 
implementation  which  is  directly  compatable  with  all  X25 
implementations  (including  itself).  Moreover,  a  complete  set 
of  choices  has  been  proposed  to  ensure  that  a  specific 
implementation  has  this  feature.  In  order  to  make  level  3 
error  notification  complete  and  consistent,  we  have  proposed 
an  extension  to  the  protocol  to  deal  with  the  case  where  the 
logical  channel  number  is  considered  invalid. 
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IV.  NETWORK  INTERCONNECTION 
4 . 1  Introduction 

Clearly  a  large  part  of  our  work  is  concerned  with  network 
interconnection.  In  order  to  develop  the  configuration 
plan  of  Section  2.3,  we  have  gone  through  many  iterations 
(14-17).  It  would  be  mainly  of  historical  interest  to 
trace  through  why  we  are  proposing  now  the  configuration 
of  Section  2.3. 

Our  work  has  had  many  aspects.  Several  papers  (e.g.  18) 
discussed  the  connection  of  campus  networks  to  public 
networks.  In  a  more  major  published  paper,  we  collaborated 
with  V.G.  Cerf  in  giving  a  general  review  of  the  issues 
in  interconnecting  packet  networks.  This  paper  is  reproduced 
as  Section  4.2.  In  another  paper,  we  collaborated  with 
G.R.  Grossman  in  analysing  the  issues  raised  by  the  PTT  X25 
recommendation  for  the  connection  of  Public  Packet  Data 
Networks;  this  paper  is  reproduced  as  Section  4.3. 

Another  contribution  (with  C.  Sunshine)  (19)  was  the  prepar¬ 
ation  of  an  IFIP  document  on  the  same  subject  to  the  CCITT. 

High  level  protocols  over  concatenated  networks  have  concer¬ 
ned  us  greatly.  We  have  put  in  a  major  effort  in  trying 
to  define  the  services  that  are  required  (20)  and  then  setting 
out  to  meet  the  requirements.  A  key  problem  here  is  the 
provision  of  bulk  data  transfer  facilities.  One  activity 
here  has  been  the  mapping,  on  a  PDP-9,  between  the  ARPANET 
and  EPSS  methods  of  operation.  This  development  is  well 
advanced  (21),  and  should  be  completed  in  1979.  A  much 
more  significant  development  has  been  to  start  implementing 
a  Network  Independent  File  Transfer  Protocol  (NIFTP)  on 
an  ARPANET  Host,  in  order  to  stage  file  transfers  over 
concatenated  Virtual  Calls.  Some  details  of  specific  appli¬ 
cations  are  given  in  Working  Papers  (22).  A  more  general 
discussion  of  the  problems  involved  is  given  as  Section  4.4. 
We  propose  to  evaluate  the  differences  between  the  protocol 
translation  of  (21)  and  the  file  staging  of  Section  4.4. 

In  June  there  was  an  EPSS  open  day  at  UCL;  as  part  of  this 
we  demonstrated  Satnet  experiments  through  a  concatenation 
of  ARPANET,  EPSS  and  Satnet.  A  discussion  of  that  demonstr¬ 
ation,  which  can  be  made  at  any  time  since  it  uses  our  stan¬ 
dard  service  offerings  -  is  given  in  Section  4.5. 


4.2  Issues  in  Packet-Network  Interconnection 
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I.  Introduction 
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switching  technology  For  .single  otganiralions,  these  data 
networks  arc  often  private  ones,  huilt  with  a  technology 
optuiii/ed  to  the  specific  application  I'oi  communication 
hetween  organirations,  these  networks  are  Ireing  set  up  hy 
licensed  carriers.  In  North  Aiiienca,  there  are  many  such 
licensed  carriers,  eg.  Ti  l  KNI  T  HI.  OATAPAC  121,  and 
TYMNFT  13|  In  the  rest  of  the  world,  the  Pivst.  Telegraph, 
and  Telephone  Authoiity  tPTIT  in  each  country  has  a  neat 
monopoly  on  such  services,  special  public  data  networks 
being  set  up  in  these  countnes  include  TRANSPAC  |.'l  in 
France,  FURONFT  lo|  for  inter-Fiiropean  traffic,  HDX  |7| 
in  Japan,  FOS  18)  in  the  Federal  Republic  of  (>erniany,  and 
the  Nordic  Public  Oats  Network  (NPDN.  Idllm  Scandinavia 
These  public  data  networks  are  considered  in  greater  detail 
in  other  references  (e  g  ,  1 10|>|  I  2)  1.  Mast  of  the  above  net¬ 
works  use  packet  switching  technology,  some  of  them.  eg,. 
FD,S  and  the  NPI1N,  do  not  do  so  yet,  but  may  do  so  in  the 
future.  In  some  ca.scs  s|iecial  data  networks  have  been  authev 
rued  for  specific  communities,  e  g  .  .SITA  |  l.l|  for  the  airlines, 
and  SWIFT  1 141  (or  the  banks  In  addition  many  private  net¬ 
works  have  been  set  up  among  individual  organi/ations,  and 
experimental  networks  of  different  technologies  have  been 
developed  also,  eg.  ARPANFT  (151.  |lbl.  CYCV  ADFS 
1171,  FTHFRNFT  1181,  SPYDFR  ll^l,  PRNFT  1201, 1211 
and  SATNFT  1221 
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It  IS  a  common  user  requirement  that  a  single  terminal  anil 
access  port  should  be  able  to  access  any  computing  resource 
the  user  may  desire  even  if  the  resource  is  on  another  data 
network.  From  this  requirement,  there  is  a  clear  user  need  to 
have  data  networks  connected  together  By  the  same  token, 
the  providers  of  data  network  services  would  like  to  have  then 
networks  used  as  intensively  as  possible,  thus  they  also  have  a 
strong  motivation  to  connect  their  data  networks  to  others 
As  a  result  of  these  considerations,  there  has  been  a  high 
recent  interest  in  the  issues  arising  in  the  connection  of  data 
networks  1231-1261,  (321 

From  the  user  viewpoint,  the  requirement  for  interconnec¬ 
tion  of  data  networks  is  independent  of  the  network  tech¬ 
nology.  From  the  implementation  viewpoint,  there  can  be 
some  considerable  complications  in  connecting  networks  of 
widely  different  technologies  such  as  circuit-switched  and 
datagram  packet-switched  networks  (these  terms  are  explained 
below).  On  the  whole  we  will  consider  only,  in  this  paper,  the 
interconnection  of  packet-switched  data  networks  In  many 
cases,  however,  the  arguments  will  be  equally  valid  for  the  inter¬ 
connection  of  packet-switched  to  circuit-switched  networks. 

Network  interconnection  raises  a  great  many  technical,  legal, 
and  political  questions  and  issues.  The  technical  issues  gen¬ 
erally  revolve  around  mechanisms  for  achieving  interconnec¬ 
tion  and  their  performance.  How  can  networks  be  intercon¬ 
nected  so  that  packets  can  flow  in  a  controllable  way  from  one 
net  to  another?  Should  all  computer  systems  on  all  nets  be 
able  to  communicate  with  each  other?  How  can  this  be 
achieved?  What  kind  of  performance  can  be  achieved  with  a 
set  of  interconnected  networks  of  widely  varying  internal 
design  and  operating  characteristics?  How  are  terminals  to  be 
given  access  to  resources  in  other  networks?  What  protocols 
are  required  to  achieve  this?  Should  the  protocols  of  one  net 
be  translated  into  those  of  another,  or  should  common  proto¬ 
cols  be  defined?  What  kinds  of  communication  protocol 
standards  are  needed  to  support  efficient  and  useful  inter¬ 
connection?  Who  should  take  responsibility  for  setting 
standards? 

The  legal  and  political  issues  are  at  least  as  complex  as  the 
technical  ones.  Can  private  networks  interconnect  to  each 
other  or  must  they  do  so  through  the  mediation  of  a  public 
network?  How  is  privacy  to  be  protected?  Should  there  be 
control  over  the  kinds  of  data  which  move  from  one  net  to 
another?  Are  there  international  agreements  and  conventions 
which  might  be  affected  by  international  interconnection  of 
data  networks?  What  kinds  of  charging  and  accounting 
policies  should  apply  to  multinetwork  traffic?  How  can  faults 
and  errors  be  diagnosed  in  a  multinet  environment?  Who 
should  be  responsible  for  correcting  such  faults?  Who  should 
be  responsible  for  maintaining  the  gateways  which  connect 
nets  together? 

We  cannot  possibly  answer  all  of  these  questions  in  this 
paper,  but  we  deal  with  many  of  them  in  the  sections  below. 

This  paper  is  divided  into  eleven  sections.  In  the  next  sec¬ 
tion  we  provide  some  definitions,  and  in  Section  III  we  ex¬ 
plore  some  of  the  motivations  for  network  interconnection. 
In  Section  IV  we  discuss  the  range  of  end-user  service  require¬ 
ments  and  choices  for  providing  multinetwork  service.  Section 
V  reviews  the  concept  of  computer-communication  protocol 
layering.  Section  VI  reviews  the  basic  interconnection  choices 
and  introduces  the  concept  of  gateways  between  nets,  proto¬ 
col  translation  and  the  impact  of  common  protocols;  it  elabo¬ 
rates  also  on  the  function  of  gateways.  Section  VII  discusses 


the  (TITT  recommendations  X.25  and  X.75  and  their  role  in 
network  interconnection  Section  VIII  describes  some  of  the 
network  interconnections  achieved  and  some  of  the  experi¬ 
ments  in  progress  Section  IX  outlines  regulatory  issues  raised 
by  network  interconnection  alternatives.  Section  X  mentions 
some  unresolved  research  questions,  and  the  final  section 
offers  some  tentative  conclusions  on  network  interconnection 
issues. 

II.  Tiu  IJkHNiTioN  Of  Terms 

The  vocabulary  of  networking  is  extensive  and  not  always 
consistent.  We  introduce  some  generic  terms  below  which  we 
will  use  in  this  paper  for  purposes  of  discussion.  It  is  impor¬ 
tant  for  the  reader  not  to  make  any  a  priori  assumptions  about 
the  physical  realization  of  the  objects  named  or  of  the  bound¬ 
ary  of  jurisdictions  owning  or  managing  them.  For  instance, 
a  gateway  (see  iielow)  might  be  implemented  to  share  the 
hardware  of  a  packet  switch  and  be  owned  by  a  packet-switch¬ 
ing  service  carrier,  alternatively  it  might  be  embedded  in  a  host 
computer  which  subscribes  to  service  on  two  or  more  com¬ 
puter  networks.  Roughly  speaking,  we  are  assigning  names  to 
groups  of  functions  which  may  or  may  not  be  realized  as 
physically  distinct  entities. 

Packet:  A  packet  of  information  is  a  finite  sequence  of  bits, 
divided  into  a  control  header  part  and  a  data  part.  The  header 
will  contain  enough  information  for  the  packet  to  be  routed 
to  its  destination.  There  will  usually  be  some  checks  on  each 
such  packet,  so  that  any  switch  through  which  the  packet 
passes  may  exercise  error  control.  Packets  are  generally 
associated  with  internal  packet-network  operation  and  are  not 
necessarily  visible  to  host  computers  attached  to  the  network. 

Datagram:  A  finite  length  packet  of  data  together  with 
destination  host  address  information  (and,  usually,  source 
address)  which  can  be  exchanged  in  its  entirety  between  hosts, 
independent  of  all  other  datagrams  sent  through  a  packet 
switched  network.  Typically,  the  maximum  length  of  a  data¬ 
gram  lies  between  1 000  and  8000  bits. 

Gateway:  The  collection  of  hardware  and  software  required 
to  effect  the  interconnection  of  two  or  more  data  networks, 
enabling  the  passage  of  user  data  from  one  to  another. 

Host:  The  collection  of  hardware  and  software  which  uti¬ 
lizes  the  basic  packet-switching  service  to  support  end-to-end 
interprocess  communication  and  user  services. 

Packet  Switch:  The  collection  of  hardware  and  software  re¬ 
sources  which  implements  all  intranetwork  procedures  such  as 
routing,  resource  allocation,  and  error  control  and  provides  ac¬ 
cess  to  network  packct-.switching  services  through  a  host/ 
network  interface. 

Protocol:  A  set  of  communication  conventions,  including 
formats  and  procedures  which  allow  two  or  more  end  points 
to  communicate.  The  end  points  may  be  packet  switches, 
hosts,  terminals,  people,  file  systems,  etc. 

Protocol  Translator:  A  collection  of  software,  and  possibly 
hardware,  required  to  convert  the  high  level  protocols  used  in 
one  network  to  those  used  in  another. 

Terminal:  A  collection  of  hardware  and  possibly  software 
which  may  be  as  simple  as  a  character-mode  teletype  or  as 
complex  as  a  full  scale  computer  system.  As  terminals  increase 
in  capability,  the  distinction  between  “host”  and  “terminal" 
may  become  a  matter  of  nomenclature  without  technical 
substance. 

Virtual  Circuit  A  logical  channel  between  source  and  desti¬ 
nation  packet  switches  in  a  packet-switched  network.  A 
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may  nol  hi-  visihlc  to  ihr  suhscrihcr  l‘av.ket\  sent  on  a  virtual 
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/’/  /'  Icchnically  PIT  stands  for  Post,  telegraph,  and  Tele¬ 
phone  Authoiily.  this  authority  has  a  diflerent  form  in  differ 
ent  countries  In  this  paper,  by  PTV  we  mean  merely  the 
authority  (or  authorities!  licensed  in  each  country  to  offer 
public  data  transmission  services 

We  have  attempted  to  make  these  definitions  as  noncontro- 
versial  as  possible  P'or  example,  in  the  definition  of  packet 
switch,  we  alluded  to  a  host /net  work  interface  The  reader 
should  not  assume  that  subscriber  services  are  limited  to  those 
offered  through  the  host/nefwork  interface  The  packet¬ 
switching  carrier  might  also  offer  host-based  services  and 
terminal  access  mechanisms  as  additional  subscriber  services. 

Ill  Till.  MOTIVATINi:  FoRt  KS  IN  TMK 
Intkrchnnh'tidn  Ob  Data  Nktworks 

In  the  introduction,  we  mentioned  that  there  was  a  strong 
interest,  among  both  the  users  and  suppliers  of  data  senvees,  in 
the  interconnection  of  data  networks.  However,  the  technical 
interests  of  the  different  parties  are  not  identical.  The  end 
user  would  merely  like  to  be  able  to  access  any  resources  from 
a  single  terminal,  with  a  single  access  port,  as  economically 
as  possible  according  to  his  own  perfiirmance  entena  A 
Public  Carrier,  or  PTT,  has  a  strong  motivation  to  connect  its 
network  to  other  PTT's.  As  in  the  telephone  system,  the 
concept  of  all  subscribers  being  accessible  through  a  single 
Public  Data  Service,  is  considered  highly  desirable,  however 
the  different  PTT’s  may  have  restricted  geographic  coverage, 
or  only  a  specific  market  penetration 

The  motivation  of  the  PTT's  to  interface  to  private  networks 
IS  weaker  and  more  complex.  They  always  provide  facilities 
to  attach  single  terminals,  where  a  terminal  may  be  a  complex 
computer  system,  they  are  often  not  interested,  at  present,  in 
making  any  special  arrangements  when  the  ''terminal"  is  a 
whole  computer  network.  The  operators  of  private  networks 
often  have  a  vital  interest  in  connecting  their  networks  to 
other  private  networks  and  to  the  public  vines,  l-ven  though 
in  many  cases  the  bulk  of  its  traffic  is  internal  to  the  private 
network,  which  is  why  it  was  set  up  in  the  fint  place,  there  is 
usually  a  vital  need  to  access  resources  not  available  on  that 
network.  The  regulatory  limitations  often  imposed  on  the 
method  of  interconnection  of  private  networks  are  discussed 
in  Section  IX.  In  some  countries,  it  is  not  permitted  to  build 
private  networks  using  leased  line  services,  but  intrabuilding 
networks  may  be  permitted.  Interconnection  of  such  local 
networks  to  public  networks  may  play  a  crucial  role  in  making 
the  local  network  useful. 

To  date  the  PTT"s  have  tried  to  standardize  on  access  pro¬ 
cedures  for  their  Public-Packet  Data  Services.  The  standardiza¬ 
tion  has  taken  place  in  the  International  Consultative  Commit¬ 
tee  on  Telegraphy  and  Telephony  (called  CCITT)  in  a  set  of 
recommendations  called  X..I,  X.2S,  X.28,  and  \.2^  (1271- 
1 2d| ).  Not  all  PTT's  have  such  forms  of  access  yet,  but  most 
of  the  industrialized  nations  in  the  West  are  moving  in  this 
direction.  Thu  senes  of  recommendations  is  discussed  in 
much  more  detail  in  Section  VI;  it  does  not  pay  special  atten¬ 
tion  to  the  attachment  of  private  networks  ((.Ml,  132U,  but 
the  recommendations  are  themselves  expected  to  change  to 
meet  this  requirement.  The  PTT's  are  agreeing  on  a  set  of  inter¬ 
face  recommendations  and  procedures  called  X.7.S  (33],  to 
connect  their  networks  to  each  other,  so  far  this  interface 
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ptocedurc  (and  its  corrcspunding  hardware!  is  not  intended 
to  be  provided  to  private  networks. 

While  most  PTT's  have  prelerred  to  ignore  the  technical 
implications  of  the  attachment  of  private  networks  to  the 
public  ones,  most  private  network  operators  vannot  ignore 
this  requirement  They  are  otten  motivated  to  add  some  extra 
"Foreign  F'xchange"  capability  as  an  afterthought,  with  mini¬ 
mum  change  to  their  intranetwoik  procedures,  this  approach 
can  be  successful  up  to  a  point,  but  will  usually  be  limited  by 
the  lack  of  high-level  procedures  between  the  different  net¬ 
works.  These  high-level  procedures  have  not  yet  been  con¬ 
sidered  by  CCITT,  but  it  has  been  proposed  that  CCITT  Study 
Clroup  Vll  investigate  higti-level  procedures  and  architectural 
models,  in  cooperation  with  the  investigation  of  "open  system 
architectures"  by  Technical  Committee  d?.  Sub-Committee 
lb  of  the  International  Standards  Organisation  (ISO).  This 
subject  IS  also  considered  later  in  this  paper,  in  Section  VI. 

An  aim  of  these  standardization  exercises  is  to  ensure  that 
both  inanufacturei  and  user  implementations  of  network 
resources  can  communicate  with  each  other  through  single 
private  or  public  data  networks.  A  consequence  should  he 
that  the  resources  are  also  compatibly  accessible  over  con¬ 
nected  data  networks 

Depending  on  the  applications  and  spatial  distnbution  of 
subscribers,  the  preferred  choice  of  packet-switching  medium 
will  vary.  Intrabuilding  applications  such  as  electronic  office 
services  may  be  most  economically  provided  through  the  use 
of  a  coaxial-packet  cable  system  such  as  the  Xerox  F'THF'RNl-T 
(181  and  l.CSNF'T  (bA) .  or  twisted  pair  rings  such  as  DCS 
(34) ,  coupled  with  a  mix  of  self-contained  user  computers 
(e.g.,  intelligent  terminals  with  substantial  computing  and 
mem.iry  capacity)  and  shared  computing,  storage,  and  input- 
output  facilities  Larger  area  regional  applications  might  best 
employ  shared  video  cables  13,'l  or  packet  radios  (  201 .1211 
for  mobile  use  National  systems  might  be  composed  of  a  mix 
ture  of  domestic  satellite  channels  and  conventional  leased- 
line  services.  International  systems  might  use  point-to-point 
links  plus  a  shared  communication  satellite  channel  and  multi¬ 
ple  ground  stations  to  achieve  the  most  cost-effective  service. 

A  consequence  of  the  wide  range  of  technologies  which  are 
optimum  for  different  packet  switching  applications  is  that 
many  different  networks,  both  private  and  public,  may  co-exist 
A  network  interconnection  strategy,  if  properly  designed,  will 
permit  local  networks  to  be  optimized  without  sacrificing  the 
pos.sibility  of  providing  effective  internetwork  services.  The 
potential  economic  and  functional  advantages  of  local  net¬ 
works  such  as  FTMFRNHT  or  IX'S  will  lead  naturally  to  pn 
vate  user  networks.  .Such  private  network  developments  arc 
analogous  to  telephone  network  pnvate  automated  branch 
exchanges  (PABX)  and  represent  a  natural  consequence  of 
the  marriage  of  computer  and  telecommunication  technology. 

Two  further  developments  can  be  expected.  First,  organiza¬ 
tions  which  are  dispersed  geographically,  nationally,  or  inter 
nationally,  will  want  to  interconnect  these  private  networks 
both  to  share  centralized  resources  and  to  effect  intraorganiza¬ 
tion  electronic  mail  and  other  automated  office  services. 
Second,  there  will  be  an  increasing  interest  in  interorganization 
intea'onnections  to  allow  automated  procurement  and  finans'ial 
transaction  services,  for  example,  to  be  applied  to  interorgani¬ 
zation  affairs. 

In  most  countries  where  private  networks  arc  permitted, 
interorganization  telecommunication  requires  the  involvement 
of  a  PTT.  Hence  the  most  typical  network  interconnection 
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scenario!!  will  in\i>lvc  three  or  four  networks  Within  one  na- 
tional  adiMinistrjliuii  the  private  nets  of  different  oriianir.a- 
tions  wjl  be  iiiterconnei'ted  through  a  public  network  Inter¬ 
national  interconnections  will  involve  at  least  two  public 
networks  We  will  return  to  this  topic  in  Section  VI 

In  addition  to  perniittinK  locally  optinii/ed  networks  to  he 
interconnected,  a  network  interconnection  strategy  should 
also  support  the  luadual  introduction  of  new  networking 
technology  into  existing  systems  without  requinng  simul¬ 
taneous  global  change  throughout  This  consideration  leads 
to  the  conclusion  that  the  public  data  networks  should  sup¬ 
port  the  most  important  user  requirements  for  internet  service 
from  the  outset.  If  this  were  the  case,  then  changes  in  net¬ 
work  technology  which  require  a  iiiultinetwork  system  during 
phased  transition  would  not,  a  priori,  have  to  affect  user 
services. 

IV.  PROVtStllN  Ol  I'ND-USKR  MllLTlNI-.TWORK  SeRVICKS 

Ihe  ultimate  choice  of  a  network  interconnection  strategy 
will  be  strongly  affected  by  the  types  of  user  services  which 
must  be  supported.  It  is  useful  to  consider  the  range  of  exist¬ 
ing  and  foreseeable  user  service  requirements  without  regard 
for  the  precise  means  by  which  these  requirements  are  to  be 
met.  We  will  leave  for  discussion  in  subsequent  sections  the 
choice  of  supporting  the  various  services  within  or  external  to 
the  packet-switched  network.  The  types  of  service  discussed 
below  are  general  requirements  for  network  facilities.  For  this 
reason  they  also  should  be  supported  across  interconnected 
networks. 

Most  of  the  currently  prevalent  computer-communication 
services  fall  into  four  categories. 

1 )  terminal  access  to  time-shared  hast  computers. 

2)  remote  job  entry  services  (RJF); 

.t)  bulk  data  transfer, 

41  transaction  priKCSsing. 

The  time-sharing  and  transaction  services  typically  demand 
short  network  and  hast  response  times  but  modest  bandwidth. 
The  RJF  and  file  transfer  services  more  often  require  high 
amounts  of  data  transfer,  but  can  tolerate  longer  delay.  Some 
networks  were  designed  to  support  primarily  terminal  service, 
leaving  RJF  or  file  transfer  services  to  be  supported  by  dedi¬ 
cated  leased  lines.  Packet-switching  techniques  permit  both 
types  of  service  to  be  supported  with  common  network 
resources,  leading  to  verifiable  economies.  However,  bulk 
data  transfer  requires  increasingly  higher  throughput  rales  if 
delivery  delays  are  to  be  kept  constant  as  the  amoivnt  of 
data  to  be  transferred  increases. 

As  distributed  operating  systems  become  more  prevalent, 
there  will  be  an  increased  need  for  host-to-host  transaction 
services.  A  prototypical  example  of  such  a  system  is  found  in 
the  DARPA  National  Software  Works  |4],  136).  In  such  a 
system,  small  quantities  of  control  information  must  be  ex¬ 
changed  quickly  to  coordinate  the  activity  of  the  distributed 
components.  Broadcast  or  multidestination  services  will  be 
needed  to  support  distributed  file  systems  in  which  informa¬ 
tion  can  be  stored  redundantly  to  improve  the  reliability  of 
access  and  to  protect  against  catastrophic  failures. 

Transaction  services  are  also  finding  application  in  reserva¬ 
tion  systems,  credit  verification,  point  of  sale,  and  electronic 
funds-transfer  systems  in  which  hundreds  or  thousands  of 
terminals  supply  to,  or  request  of,  hosts  small  amounts  of 
information  at  random  intervals.  Real-time  data  collection  for 
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weather  analysis,  ground  and  air  traffic  control,  and  meter 
reading,  for  example,  also  fall  into  this  category. 

More  elaborate  user  requirements  can  be  foreseen  as  elec¬ 
tronic  mail  facilities  propagate.  Multiple  destination  address¬ 
ing  and  end-to-end  encryption  for  the  protection  of  privacy 
as  well  as  support  for  text,  digitized  voice,  and  facsimile  mes¬ 
sage  transmission  are  all  likely  requirements.  Electronic  tele¬ 
conferencing  using  mixtures  of  compressed  digital  packet 
speech,  videographics,  real-time  cursors  (for  pointing  at  video 
images  under  discussion),  and  text  display  will  give  rise  to  re¬ 
quirements  for  closed  user  groups  and  time-synchronized 
mixes  of  transaction-like  (e.g.,  for  cursor  tracking  and  packet 
speech)  and  reliable  circuit-like  services  (e.g.,  for  display 
management). 

Reliability  and  rapid  response  will  be  increasingly  important 
as  more  and  more  computer-based  applications  requiring  tele¬ 
communications  are  integrated  into  the  business,  government, 
military,  and  social  fabric  of  the  world  economy.  The  more 
such  systems  are  incorporated  into  their  daily  activities,  the 
more  vulnerable  the  subscribers  are  to  failures.  Reliability 
concerns  lead  to  the  lequirement  for  redundant  alternatives 
such  as  distributed  file  systems,  richly  connected  networks, 
and  substantial  local  processing  and  storage  capability.  These 
trends  increase  the  need  for  networking  to  share  common 
hardware  and  software  resources  (and  thus  reduce  their  mar¬ 
ginal  cost),  to  support  remote  software  maintenance  and  de¬ 
bugging.  and  to  support  intra-  and  inter-organizational  infor¬ 
mation  exchange. 

We  have  described  the  end-user  services  required  across  one 
or  more  data  networks.  We  have  carefully  refrained  from  dis¬ 
cussing  which  services  should  be  provided  in  the  data  network, 
and  which  should  be  provided  in  the  hosts.  Here  the  choice 
in  single  networks  will  depend  on  the  network  technology  and 
the  application  requirements.  For  example,  in  a  network  using 
a  broadcast  technology  such  as  FTHF'RNFT  or  the  SATNET, 
multidestination  facilities  may  well  be  incorporated  in  the  data 
network  itself.  In  typical  store- and-forward  networks,  this 
feature  might  be  provided  at  the  host  level  by  the  transmission 
of  multiple  copies  of  packets.  This  example  highlights  im¬ 
mediately  the  difficulty  of  using  sophisticated  services  at  the 
data  network  level  across  concatenated  networks.  If  X,  A, 
and  r  are  data  networks  connected  as  in  Fig.  I .  and  A  and  (' 
but  not  B  support  broadcast  or  real-time  features,  it  is  very 
difficult  to  provide  them  across  the  concatenation  of  X ,  A,  and 

r. 

The  problem  of  achieving  a  useful  set  of  internetwork  ser¬ 
vices  might  be  approached  in  several  ways,  as  follows. 

I)  Require  all  networks  to  implement  the  entire  range  of 
desired  services  (e.g.,  datagram,  virtual  circuit,  broadcast,  real- 
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timr.  ell.  I,  aiiJ  then  attempt  to  support  these  services  acros.s 
the  ^teisays  between  the  networks. 

J)  Kcgiiire  all  networks  to  implement  only  the  most  basic 
services  tet;..  ilatatirani  or  virtual  circuit),  support  these  ser¬ 
vices  across  Kateways,  and  rely  on  the  subscriber  to  imple¬ 
ment  all  other  services  end-to-end. 

.))  Allow  the  subscriber  to  identify  the  services  which  he 
desires  and  provide  error  indications  if  the  networks  involved, 
or  the  iiateways  between  them,  cannot  provide  the  desired 
services. 

4)  Allow  the  suKsenber  to  specify  the  internetwork  route  to 
be  followed  and  depend  on  the  subscriber  to  decide  which 
concatenation  of  services  arc  appropriate  and  what  end-to-end 
protiKols  are  needed  to  achieve  the  ultimately  preferred  class 
of  service 

5)  Provide  one  set  of  services  for  local  use  within  each  net¬ 
work  and  another,  possibly  different  set  for  internetwork 
use 

The  five  choices  above  are  by  no  means  exhaustive,  and,  in 
fact,  only  scratch  the  surface  of  possibilities.  Nothing  has 
been  said,  thus  far.  about  the  compatibility  of  various  levels 
of  communication  protocols  which  exist  within  each  network, 
within  subsenber  equipments,  and  within  the  logical  gateway 
between  networks.  To  explore  these  issues  further,  it  will  be 
helpful  to  have  a  model  of  internetwork  architecture,  taking 
into  account  the  common  principle  of  protocol  layering  and 
the  vanous  possible  choices  of  interconnection  strategy  which 
depend  upon  the  protiKol  layer  at  which  the  networks  are 
interfaced  We  consider  this  in  the  next  section. 

V.  LAYt.RH)  PrOTCX'DL  CONI’HTS 

Both  to  provide  services  in  single  networks,  and  to  compare 
the  capabilities  of  different  networks,  a  very  useful  concept 
in  networking  is  protocol  layering.  Various  services  of  increas¬ 
ing  capability  can  be  built  one  on  top  of  the  other,  each  using 
the  facilities  of  the  service  layer  below  and  supporting  the 
facilities  of  the  layer  above.  A  thorough  tutorial  on  this  con¬ 
cept  can  be  found  in  the  paper  by  l*ou/.in  and  /.immermann  in 
this  cssue  l.<7| .  We  give  some  specific  examples  below  of  layer¬ 
ing  as  a  means  of  illustrating  the  scope  of  services  and  inter¬ 
faces  to  he  found  in  packet  networks  today  and  some  of  the 
problems  encountered  in  offering  services  across  multiple 
networks 

fable  I  offers  a  very  generic  view  of  a  typical  protocol 
hierarchy  in  a  store-and-forward  computer  network,  including 
layers  usually  tound  outside  of  the  communication  network 
Itself  There  are  several  complications  to  the  use  of  genenc 
protocol  layering  to  study  network  interconnection  is.sues 
Chief  among  these  is  that  networks  do  not  all  contain  the  same 
elements  of  the  generic  hierarchy.  A  second  complication  is 
that  some  networks  implement  service  functions  at  different 
protocol  layers.  For  instance,  virtual  circuit  networks  imple¬ 
ment  an  end/end  subscriber  virtual  circuit  in  their  intranet, 
end/end  level  protocol.  Finally,  the  hierarchical  ordering  of 
functions  is  not  always  the  same  in  all  networks  For  instance, 
TYMNFT  places  a  terminal  handling  protocol  within  the  net¬ 
work  access  layer,  so  that  hosts  look  to  each  other  like  one  or 
more  terminals.  Figs.  2-7  illustrate  the  functional  layering 
of  some  different  networks.  It  is  important  to  note  how  the 
functions  vary  with  the  choice  of  transmission  medium 

.1  A  TiniiNh  T 

In  Fig  2.  we  represent  the  Xerox  FIIII  KNFr  protocol 
hierarchy  The  basic  link  control  mechanism  is  the  ability  of 
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the  interface  device  to  detect  conflict  on  a  shared  coaxial  cable 
If  a  transmitting  interlace  detects  that  another  interface  is 
also  transmitting,  it  immediately  aborts  the  transmission 
Hosts  attached  to  the  network  interface  present  datagrams  to 
be  transmitted  and  are  told  if  the  datagram  was  aborted 
Datagrams  can  be  addressed  to  specific  interfaces  or  to  all  of 
them  The  end.end  subscriber  layer  of  protocol  is  split  into 
two  parts  a  reliable  datagram  protm'ol  in  which  each  data 
glam  IS  reliably  delivered  and  sepaialely  acknowledged,  .md 
a  stream  proUwol  which  can  be  thought  of  as  a  virtual  circuit 
This  split  IS  pos.sible,  m  part,  because  there  is  a  fairly  large 
maximum  datagram  si/e  labout  .^00  bytes)  so  that  user  appli 
cations  can  send  daiagiams  without  having  Ii)  Iragmeiit  and 
reassemble  them  This  makes  the  datagram  service  useful  for 
many  applications  which  might  otherwise  have  to  use  the 
stream  protocol  All  higher  level  protocols,  such  as  Virtual 
rerminal  and  File  franster.  are  carried  nut  in  the  hivsts 

H  ARrAShT 

The  ARPANFT  protiKol  hierarchy  is  shown  in  Fig  .t  The 
basic  link  control  between  packet  switches  treats  the  physical 
link  as  eight  independent  virtual  links  This  increases  effec 
tive  throughput,  but  does  not  neces»-arily  pieserxc  the  ordei 
in  which  packets  were  originally  introduced  into  the  network 
The  intranet  node-tevnode  protivols  deal  with  adaptive  tout 
ing  decisions,  store-and-forward  service,  and  congestion  con 
trol  Hivsts  have  the  option  of  either  passing  mes.sages  tup  to 
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HOt>y  bics  of  text)  across  the  host/network  interface,  which 
will  be  delivered  in  sequence  to  the  destination,  or  passing 
datagrams  (up  to  1008  bits  of  text)  which  are  not  necessanly 
deliverej  m  sequence  The  user’s  network  access  interface  is 
datagrani-like  in  the  sense  that  no  circuit  setup  exchange  is 
needed  even  to  activate  the  sequenced  message  service.  In 
effect,  this  service  acts  like  a  permanent  virtual  circuit  over 
which  a  sequence  of  discrete  messages  are  sent  For  the 
sequenced  messages,  there  is  exactly  one  virtual  circuit  main¬ 
tained  for  each  host/host  pair.  In  fact,  these  virtual  circuits 
are  set  up  dynamically  and  terminated  by  the  source/destina- 
(ion  packet  switches  so  as  to  improve  resource  utilization 
Id«l,  |62|. 

The  end/end  subscriber  layer  of  ARPANV.T  contains  two 
main  protocols  Network  Control  Protocol  ( NCP,  |3‘>|,  (401) 
and  Transmission  Control  Protocol  (TCP,  (25| ).  NCP  was  the 
first  interprocess  communication  protiKol  built  for  ARPANFT. 
It  relics  on  the  sequenced  message  service  provided  by  the  net¬ 
work  and  derives  multiple  virtual  circuits  between  pairs  of 
hosts  by  multiplexing  The  TCP  can  use  either  the  sequenced 
mrs.sagr  service  or  the  datagram  service  It  docs  its  own 
sequencing  and  end/end  error  control  and  denves  multiple 
virtual  circuits  through  extended  addressing  and  multiplexing. 
TCP  was  designed  for  operation  in  a  multinet  environment  in 
which  the  only  service  which  reasonably  could  be  expected 
was  an  unreliable,  unsequenced  datagram  service. 

To  support  experiments  in  packetized  voice  communication, 
two  protocols  were  developed  for  use  on  the  ARPANFT.  The 
Network  Voice  Protocol  (NVP)  and  Network  Voice  Confer¬ 
encing  Protocol  (NVCP)  use  the  datagram  service  to  achieve 
verv  low  delay  and  interarrival  time  vanance  in  support  of 
digital,  compres.sed  packet  speech  (mote  on  these  protocols 
may  be  found  in  (4I|),  The  NVP  could  be  considered  the 
basis  for  a  generic  protocol  which  could  support  a  variety  of 
real-time,  end/end  user  applications. 

The  higher  level  utility  protocols  such  as  terminal/host 
protocol  (TFLNIiT,  (40).  (42))  and  file  transfer  protocol 
(FTP,  (40) ,  (421  )u»e  virtual  circuits  provided  by  NCP  or  TCP. 
The  FTP  requires  one  live  interactive  stream  to  control  the 
data  transfer,  and  a  second  for  the  data  stream  itself.  Yet 
higher  level  applications  such  as  electronic  mail  and  remote 
job  entry  (RJF.  (401,  (42) )  use  mixtures  of  TELNKT  and 
FTP  to  effect  the  service  desired.  These  protocols  are  usually 
put  into  the  hosts.  There  is  one  anomaly,  which  occurs  in 
many  networks  Because  terminal  handling  is  required  so 
frequently,  a  Terminal  Interface  Message  Processor  (TIP,  (43]) 
was  built  This  device  is  physically  integrated  with  the  packet 
switch  (IMP,  (380,  it  includes  also  the  NCP  and  TELNET 
protocols. 
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C  TYMNKT 

TYMNET  (see  Fig  4)  is  one  of  the  oldest  of  the  networks  in 
the  collection  described  here  (3)  Stnctly  speaking,  it  oper¬ 
ates  rather  differently  than  other  packet-switched  networks, 
because  the  frames  of  data  that  move  from  switch  to  switch 
are  disassembled  and  reassembled  in  each  switch  as  an  integral 
part  of  the  store-and-forward  operation.  Nevertheless,  the  net¬ 
work  benefits  from  the  asynchronous  sharing  of  the  circuits 
between  the  switches  in  much  the  same  way  that  more  typical 
packet-switched  networks  do.  The  network  was  designed  to 
support  remote  terminal  access  to  time-shared  computer  re¬ 
sources.  The  basic  service  is  the  transmission  of  a  stream  of 
characters  between  the  terminal  and  the  serving  host.  A 
frame  is  made  up  of  one  or  more  blocks  of  characters,  each 
block  labeled  with  its  source  terminal  identifier  and  length. 
The  switch-to-switch  layer  of  protocol  disassembles  each  frame 
into  Its  constituent  blocks  and  uses  a  routing  table  to  deter¬ 
mine  to  which  next  switch  the  block  should  be  sent.  Blocks 
destined  for  the  same  next  switch  are  batched  together  in  a 
frame  which  is  checksummed  and  sent  via  the  link  control 
procedure  to  the  next  switch  Batching  the  blocks  reduces 
line  overhead  (the  blocks  share  the  frame  checksum)  at  the 
expense  of  more  CPU  cycles  in  the  switch  for  frame  dis¬ 
assembly  and  reassembly. 

The  protocol  between  TYMNET  switches  also  includes  a 
flow  control  mechanism  which,  because  of  the  fixed  routes, 
can  be  used  to  apply  back  pressure  all  the  way  back  to  the 
traffic  source.  This  is  not  precisely  an  end-to-end  flow  control 
mechanism,  but  a  hop-by-hop  back  pressure  strategy.  Charac¬ 
ter  blocks  are  kept  in  sequence  along  the  fixed  routes  so  that 
no  rescquencing  is  required  as  they  exit  from  the  network  at 
their  destinations.  The  network  interface  is  basically  a  virtual 
circuit  designed  to  transport  character  (reams  between  a 
host  ard  a  terminal  The  same  virtual  circuits  can  be  used  to 
transport  character  streams  between  hosts,  which  look  to  each 
other  like  a  collection  of  terminals.  Above  the  basic  virtual 
circuit  service,  is  a  special  echivhandling  protocol  which 
allows  the  host  and  the  terminal  handler  m  the  ’’remote 
TYMSAT”  to  coordinate  the  echoing  of  the  characters  typed 
by  a  user. 

D.  FTTSetworkt 

Many  PTT  networks,  e  g.,  TELNET,  TRANSPAC,  DATA- 
PAC,  and  EURONET  use  a  particular  network-access  protocol, 
X.25  (28],  (29]  (see  Fig.  S).  This  protocol  has  been  recom¬ 
mended  by  the  CCITT  for  public  packet-switched  data  net¬ 
works.  X.2S  is  a  three-part  protocol  consisting  of  a  hardware 
electrical  interface.  X.2I  (44),  the  digital  equivalent  of  the 
usual  V.24  or  F.lA-RS232r  modem  interface  (4$),  a  link 
control  procedure.  High  Level  Data  Link  Control  (HDLC, 
(46)).  and  a  packet-level  protocol  for  effecting  the  setup, 
use,  termination,  flow,  and  error  control  of  virtual  circuits. 
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In  jU  but  the  OAf  .M'At'  nrtwoik.  d  fixed  route  for  roulinit 
(idckets  throuKh  the  network  is  selected  Jt  the  time  the  virtual 
eireiiit  is  created  ‘’Permanent”  virtual  circuits  are  a  customer 
option,  if  used,  the  setup  phase  is  invoked  only  in  the  case  of 
a  network  failure  Between  source  and  destination  packet 
switches,  a  viilual  circuit  protocol  is  operated  which  imple¬ 
ments  end-t  end  flow  control  on  multiple  virtual  circuits 
between  pairs  of  packet  switches  L'p  to  40*^6  virtual  circuits 
between  pairs  of  host  ports  can  be  maintained  by  each  packet 
switch,  as  Compared  to  the  sinitle  virtual  circuit  provided  by 
.\RPANFT  (on  which  hosts  can  multiplex  their  own  virtual 
circuits).  This  choice  has  a  noticeable  impact  on  the  sub¬ 
scriber  interface  protocol  which  becomes  complicated  be¬ 
cause  the  subscriber  host  and  the  packet  switch  to  which  it 
attaches  must  maintain  a  consistent  view  of  the  state  of  each 
virtual  circuit  in  use 

To  provide  foi  echo  control,  user  commands,  code  conver¬ 
sion.  and  other  termiiiaTrelated  services,  these  networks 
implement  IX'ITT  Kecominendations  X.’K  124)  and  \  24 
(24 1  in  a  PAD  (Packet  Assembly  and  Disassembly  unit) 
These  protix’ols  sit  atovi  the  virtual  circuit  X  25  protocol  In 
order  to  serve  customers  desiring  a  terniiiial-to-hosl  service 
with  character  terminals,  such  as  is  provided  by  TYMNI-T  or 
bv  the  ARPANl’T  (through  the  TIP),  most  of  the  PTT  net¬ 
works  mentioned  are  developing  a  PAD  unit  A  matching 
X  24  (PAD  control  protocol)  layer  must  be  provided  in  hosts 
offering  to  service  terminals  connected  to  PAD’s. 

h  High  Level  Protocols 

The  X.25^\.28/X.24  protocol  hierarchy  does  not  include  an 
end/end  subscriber  or  high-level  protocol  layer  Some  cus¬ 
tomers  will,  in  fact,  implement  end-to-end  protocols  on  top 
of  the  virtual  circuit  protocol,  hut  others  may  not.  Several 
attempts  are  being  made  to  standardize  protocols  above  the 
network  acce.ss  level.  The  ARPANFT  community  has  de¬ 
veloped  a  1  iansmi.s.s:.>n  Control  Protocol  125)  for  internet¬ 
work  operation  to  rcpiac;  the  Network  Control  Program 
(NCPi  developed  early  ip.  the  ARPANI'f  pro.iect.  The  Inter¬ 
national  Federation  ol  Information  Processing  (IFTP)  has 
proposed  a  Transport  Station  through  its  Working  Croup  b.l 
on  Network  liileiconnection  1471  .  the  proposal  has  been  sub¬ 
mitted  to  the  International  Standards  Organisation  (ISO)  as 
a  draft  standard  In  addition,  other  communities,  e  g.,  the 
High  l  evel  Protocol  Working  Croup  in  the  UK,  have  devised 
protocols  for  Virtual  Packet  Terminals  (VPT,  |48|)and  File 
Transport  ProtiKol  (FTP,  |44| )  which  are  intended  to  be  net¬ 
work  independent  and  which  may  be  submitted  to  C'CITT. 
The  ISO  study  on  "open  systems  architecture"  and  the  pro¬ 
posed  similar  study  by  CCITT  Study  Group  VII  will  attempt 
to  evolve  higher  level  protocol  recommendations  for  existing 
and  future  data  networks. 


1  Ills  hrict  vui!iiii..ry  vil  ditlercni  nclwoic  proliAul  ijycrin(;> 
IV  III  no  way  comprcheiivivv.  but  iliustratcv  llie  diversity  ol 
protocol  designs  which  can  be  found  on  nets  providing  difter- 
cm  types  of  services  to  subscribers 

VI.  TkHNICAL  iNItRfONNtCTION  CtlOlCIs; 

.4  The  Issues 

Beginning  with  the  earliest  papers  dealing  with  strategies 
for  packet  network  interconnection  12.))-12bl.  |.)21,  the 
common  objective  of  all  the  proposed  methods  is  to  provide 
the  physical  means  to  access  the  services  of  a  luvst  on  one  net 
work  to  all  subscribers  (including  hosts)  of  all  the  intercon¬ 
nected  networks  Of  course,  limitations  to  this  accessibility 
are  envi.saged.  imposed  either  for  administrative  reasons  or 
by  the  scarcity  of  resources.  The  achievement  of  this  objec¬ 
tive  invanably  requires  that  data  produced  at  a  soutsc  in  one 
net  be  delivered  and  correctly  interpreted  at  the  desimationtsl 
in  another  network  In  an  abstract  sense,  this  boils  down  to 
providing  interprocess  communication  across  network  bound¬ 
aries.  Tven  if  a  person  is  the  ultimate  source  of  the  data, 
packet-switching  networks  must  interpose  some  degree  of  soft¬ 
ware  processing  between  the  person  and  the  destination  ser 
vice,  even  if  only  to  assemble  or  disassemble  packets  produced 
by  a  computer  terminal 

•A  fundamental  aspect  sif  interprocess  communication  is 
that  no  communication  can  lake  place  without  some  agreed 
conventions  The  communicating  processes  must  share  some 
physical  transmission  medium  (wire,  shared  memorv .  radio 
spectrum,  etc  ),  and  they  must  use  common  conventions  or 
agreed  upon  translation  methivds  in  order  to  successfully  ex¬ 
change  and  interpret  (he  data  they  wish  to  cciiiimunic.ilc.  One 
of  the  key  elements  in  any  network  interconnection  strategy 
IS  therefore  how  the  required  commonality  is  to  be  v'btjined 
In  some  c.ises.  it  is  enough  to  translate  one  protsvcs'l  into 
another  In  others.  protiKols  can  lie  held  in  common  among 
the  communicating  parties 

In  any  real  network  interconnection,  of  course,  a  number  ot 
secondary  objectives  will  aflect  the  choice  of  interconnection 
strategy  For  example  achievable  bandwidth,  reliability, 
robustness  lie.,  resistance  to  failiire.s),  security,  flexibility, 
accountability,  access  control,  resource  allocation  options,  and 
the  like  can  separately  and  jomdy  influence  the  choice  of 
interconnection  strategy.  Combinations  of  strategies  employ¬ 
ing  protocol  standards  and  protocol  translations  at  various 
levels  of  the  layered  piotcx'ol  hierarchy  are  .ilso  likely 
possibilities 

There  are  a  number  of  is.sues  which  must  be  rescilved  befvire 
a  coherent  network  inters’onnection  strategy  can  be  defined. 
,A  list  of  some  of  these  issues,  which  will  Ise  treated  in  moie 
detail  in  succeeding  sections,  is 

1)  level  of  interconnection, 

2)  naming,  addressing,  and  routing. 

.1)  flow  and  congestion  contnil. 

4)  accounting. 

5 )  access  control . 

6)  internet  services 

B.  (laiewitys  anJ  Levels  o/  Network  Intereonrteetton 

The  concept  of  a  gateway  is  coinnivin  to  all  network  inter¬ 
connection  strategies.  The  fundamental  role  cvf  the  gateway  is 
to  terminate  the  internal  protocols  of  each  network  to  which 
it  is  attached  while,  at  the  same  time,  providing  a  common 
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ground  across  which  data  from  one  network  can  pass  into 
another  However,  the  choice  of  functions  to  be  performed  in 
the  gateway  varies  considerably  among  different  interconnec¬ 
tion  strategies  (see  Fig.  bV  The  term  "gateway"  need  not 
imply  a  monolithic  device  which  joins  a  pair  of  networks.  In¬ 
deed.  the  gateway  may  merely  be  software  m  a  pair  of  packet 
switches  in  different  networks,  or  it  may  be  made  up  of  two 
parts,  one  in  each  network  (a  sort  of  "gateway  half").  In  the 
lattei  case,  the  two  halves  might  be  devices  separate  and 
distinct  from  the  network  packet  switches  or  might  be  inte¬ 
grated  with  them  Furthermore,  a  gateway  might  interconnect 
moie  than  two  networks  In  the  material  which  follows, 
every  attempt  has  been  made  to  avoid  any  implicit  choice  of 
gateway  implementation  It  is  worth  pointing  out,  however, 
that  the  "half  gateway"  concept  is  highly  attractive  from  both 
a  technical  and  a  purely  administrative  point  of  view.  Tech¬ 
nically,  each  half  could  terminate  certain  levels  of  protocol 
of  the  net  to  which  it  is  attached.  Administratively  each  half 
could  be  the  responsibility  of  the  network  to  which  it  belongs. 
Then  the  only  matters  (or  jurisdictional  negotiation  are  the 
physical  medium  by  which  the  half-gateways  exchange  data, 
a.id  the  format  and  protcKol  of  the  exchange. 


It  is  importanl  to  realize  that  typical  applications  may  in- 
vvilve  three  or  more  networks  Where  liKal  networks  are  used, 
they  wdl  usually  need  to  he  interconnected  to  realize  the 
benefits  of  interorganizational  data  exchange  In  most  coun 
tries,  such  interconnections  wjl  only  be  permitted  through  a 
public  network  Thus  for  a  typical  national  situation,  three 
networks  and  two  gateways  will  be  involved  in  providing  the 
desired  host-to-host  communication. 

The  international  picture  is  simiiar.  except  that  more  net¬ 
works  are  likely  to  be  involved  Shown  in  Fig  7.  the  path 
from  a  host.  .V.  on  local  network  /  .VH)  in  country  1.  passes 
through  a  public  network.  /’.Vi  -I )  in  country  -I .  through  an  in¬ 
ternational  network  IN,  tlirough  a  public  network  in 

country  H.  and  finally  through  a  local  network.  I..\’[H).  to  the 
destination  host,  O.  There  are  four  internetwork  gateways 
involved.  It  IS  this  model  inviilving  multiple  gateways  that 
guides  us  away  from  network  interconnection  methods  which 
rely  on  the  source  and  destination  hosts  being  in  adjacent 
networks  connected  by  the  mediation  of  a  single  gateway. 

I)  Common  Subnet  Technology  I  Packet  Level  Intercon¬ 
nection)  The  level  at  which  networks  are  interconnected  can 
be  determined  by  the  protocol  layers  terminated  by  the  gate¬ 
way.  For  example,  if  a  pair  of  identical  networks  were  to  be 
interconnected  at  the  interpacket-switch  level  of  protocol, 
we  might  illustrate  the  gateway  placement  as  shown  in  Fig.  8. 
Here  the  "gateway"  may  consist  only  of  software  routines  in 
the  adjacent  packet  switches,  eg,  f\.4)  and  /\fil.  which  pro¬ 
vide  accounting,  and  possibly  readdressing  functions.  The 
contour  model  of  protocol  layer  is  useful  here  since  it  shows 
which  levels  are  common  to  the  two  networks  and  which 
levels  could  be  different  In  essence,  those  layers  which  are 
lerniinaled  by  the  gateways  could  be  different  in  each  net, 
while  those  which  are  passed  transparently  through  the  gate¬ 
way  are  assumed  to  be  common  in  both  networks.  This  net¬ 
work  interconnection  strategy  requires  that  the  internal  ad¬ 
dress  structure  of  all  the  interconnected  networks  be  common. 
If.  for  example,  addresses  were  composed  of  a  network  identi¬ 
fier,  concatenated  with  a  packet-switch  identifier  and  a  host 
identifier,  then  addressing  of  objects  in  each  of  the  networks 
would  be  straightforward  and  routing  could  be  performed  on 
a  regional  basis  with  the  network  identifiers  acting  as  the 
regional  identifiers,  if  desired  Alternatively,  two  identical 
networks  could  adopt  a  common  network  name  and  assign 
nonduplicative  addresses  to  each  of  the  packet  switches  in 
both  networks.  This  may  require  that  addresses  in  one  net¬ 
work  be  changed. 

The  strategy  described  above  might  be  called  the  "common 
subnetwork  strategy,"  since,  in  the  end,  subscribers  of  the 
newly  formed  joint  network  would  essentially  see  a  single 
network  This  strategy  does  not  rule  out  the  provision  of 
special  access  control  mechanisms  in  the  gateway  nodes  which 
could  filter  traffic  flowing  from  one  network  into  the  other. 
Similarly,  the  gateway  nodes  could  perform  special  intemet- 
woTk  traffic  accounting  which  might  not  normally  be  per 
formed  in  a  subnet  switching  node.  This  network  interconnec¬ 
tion  method  is  limited  to  those  cases  in  which  the  nets  to  be 
connected  are  virtually  identical,  since  the  gateways  must 
participate  directly  in  all  the  subnet  protocols.  The  end-to- 
end  subnet  protocols  (e  g  source/destination  packet-switch 
protocols)  must  pass  transparently  through  the  gateways  to 
permit  interactions  between  a  source  packet  switch  in  one 
net  and  a  destination  packet  switch  in  another.  The  resulting 
network  presents  the  same  network  access  interface  to  all 
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subscribers,  and  this  leads  us  to  the  next  example  which  is 
based  on  the  concept  of  a  common  network  access  interface. 

2)  Common  Nety/ork  Access  Interfaces:  If  the  subnetwork 
protocols  are  not  identical,  the  next  opportunity  to  establish 
internetwork  commonality  is  at  the  network  access  interface. 
This  is  illustrated  in  Fig.  9.  Bach  network  is  assumed  to  have 
its  own  intranet  protocols.  However,  each  network  presents 
the  same  external  interface  to  subscribers.  This  is  illustrated 
by  showing  a  common  interface  passing  through  all  hosts, 
marked  “common  network  access  interface”  in  the  figure. 

Once  again,  the  gateway  could  be  thought  of  as  software  in 
adjacent  packet  switches.  F.ach  gateway  is  composed  of  two 
halves  formed  by  linking  the  packet  switches  of  two  nets 
together.  However,  in  this  case,  the  subnetwork  protocols  are 
terminated  at  the  gateway  so  that  the  intergateway  exchange 
looks  more  like  network  access  interaction  than  a  node-to- 
node  exchange.  This  is  the  approach  taken  by  CCITT  with 
its  X.2S  packet  network  interface  recommendation  and  X.7S 
intergateway  exchange  recommendation. 

It  is  important  to  note  that  the  intergateway  interface  could 
be  similar  to  the  standard  network  access  interface,  but  it 
need  not  necessarily  be  identical. 

There  are  two  basic  types  of  network  interface  currently  in 
use:  I)  the  datagram  interface  (311;  and  2)  the  virtual  circuit 
interface  (32].  The  details  of  these  generic  interface  types 
vary  in  different  networks;  some  networks  even  offer  both 
types  of  interface.  In  some,  the  interface  to  use  may  be 
chosen  at  subscription  time;  in  othen  it  may  be  possible  for  a 
subscriber  to  select  the  access  method  dynamically. 


A  datagram  interface  allows  the  subscriber  to  enter  packets 
into  the  network  independent  of  any  other  packets  which 
have  been  or  will  be  entered.  Each  packet  is  handled  separately 
by  the  network.  A  virtual  circuit  interface  requires  an  ex¬ 
change  of  control  information  between  the  subscriber  and  the 
network  for  the  purpose,  for  example,  of  setting  up  address 
translation  tables,  setting  up  routes  or  preallocating  resources, 
before  any  data  packets  are  carried  to  the  destination.  Some 
networks  may  implement  a  fast  select  virtual  circuit  interface 
in  which  a  circuit  setup  request  is  sent  together  with  the  first 
(and  possibly  last)  data  packet.  Other  control  exchanges 
would  be  used  to  close  the  resulting  virtual  circuits  set  up  in 
this  fashion. 

It  is  essential  to  distinguish  datagram  and  virtual  circuit 
services  from  datagram  and  virtual  circuit  interfaces.  A  data¬ 
gram  service  is  one  in  which  each  packet  is  accepted  and 
treated  by  the  network  independently  of  all  others.  Se¬ 
quenced  delivery  is  not  guaranteed.  Indeed,  it  may  not  be 
guaranteed  that  all  datagrams  will  be  delivered.  Packets  may 
be  routed  independently  over  alternate  network  paths.  Dupli¬ 
cate  copies  of  datagrams  might  be  delivered. 

Virtual  circuit  service  tries  to  guarantee  the  sequenced  de¬ 
livery  of  the  packets  associated  with  the  same  virtual  circuit. 
It  typically  provides  to  the  host  advice  from  the  network  on 
flow  control  per  virtual  circuit  as  opposed  to  the  packet-by¬ 
packet  acceptance  or  rejection  typical  of  a  datagram  service. 
If  the  network  operation  might  produce  duplicate  packets, 
these  are  filtered  by  the  destination  packet  switch  before 
delivery  to  the  subscriber.  Duplicate  packet  creation  is  a 
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comiiion  ['liciii'iMcriiin  as  in  packcl-switchcd  sturc-aiid-lurward 
sYslcms  The  basic  imidc  of  operation  is  to  forward  a  packet 
to  the  next  switili  and  await  an  acknowledgment.  After  a 
timeout,  the  packet  is  retransmitted  If  an  acknowledgment  is 
lost  due  to  line  noise,  for  example,  then  two  copies  of  the 
packet  would  have  been  transmitted.  Hven  if  the  next  switch 
IS  prepared  to  filter  duplicates  out.  a  network  which  uses  adap¬ 
tive  routing  can  deliver  a  duplicate  packet  to  the  periphery  of 
the  network  For  example,  if  a  packet  switch  receives  a  packet 
successfully  but  the  line  to  the  sender  breaks  before  the  re¬ 
ceiver  can  acknowledge,  the  sender  may  send  another  copy  to 
a  different  packet  switch  Both  packet  copies  may  be  routed 
and  delivered  to  the  destination  packet  switch  where  final 
duplicate  filtering  would  be  needed  if  virtual  circuit  service  is 
being  provided. 

Some  networks  offer  both  a  datagram  and  a  virtual  circuit 
service,  some  offer  a  single  interface,  but  different  services, 
for  example,  the  AKPANET  has  a  basic  datagram  interface. 
However,  the  subnetwork  will  automatically  provide  a  se¬ 
quenced  virtual  circuit  service  (i  e..  packets  are  kept  in 
sequence  when  they  are  delivered  to  the  destination)  if  the 
packet  IS  marked  appropriately.  Otherwise,  packets  are  not 
delivered  in  sequence  nor  are  packet  duplicates  or  losses, 
except  for  line  by  link  correction,  recovered  within  the  net¬ 
work  for  nonsequenced  types  of  traffic. 

By  contrast,  TRANSPAC  offers  a  virtual  circuit  interface 
and  service  Subscribers  transmit  “call  request"  packets 
containing  the  full  destination  address  to  the  packet  switch. 
The  request  packet  is  forwarded  to  the  destination,  leaving 
behind  a  fixed  route.  The  destination  subscriber  returns  a 
"call  accepted"  packet  which  is  delivered  to  the  caller.  As  a 


result  of  this  exchange,  the  source  subscriber  has  associated  a 
"logical  channel  number"  or  U'N,  with  the  full  source- 
destination  addresses.  Thus  subsequent  packets  to  be  sent  on 
the  same  logical  channel  arc  identified  by  the  LCN  and  are 
kept  in  sequence  when  delivered  to  the  destination. 

Finally,  it  is  po.ssible  to  implement  a  datagram-like  service 
using  a  virtual  circuit  intertace.  In  this  case,  the  exchange  of 
requejr  and  accept  packets  might  be  terminated  at  the  sub¬ 
scriber's  local  packet  switch,  so  that  even  if  packets  were  not 
delivered  in  sequence  they  might  employ  abbreviated  address¬ 
ing  for  local  subscriber  and  packet-switch  interaction. 

If  network  interaction  is  to  be  based  on  a  standard  interface, 
then  agreement  must  be  reached  both  on  the  interface  and  an 
associated  service  or  services,  furthermore,  a  common  ad¬ 
dressing  system  is  needed  so  that  a  subscriber  on  one  network 
can  address  a  packet  to  a  subscriber  on  any  other  network.  A 
weaker  assumption  could  be  made  but  we  are  deliberately 
assuming  a  truly  common  service,  interface,  and  addressing 
mechanism.  We  will  return  to  this  topic  in  a  later  section. 

The  choice  of  a  standard  network  service  through  which  to 
effect  network  interconnection  has  a  primary  impact  on  the 
flexibility  of  implemcntable  network  interconnection  methods. 
We  will  consider  two  choices;  datagram  service  and  virtual 
circuit  service. 

aj  Datagram  service  as  a  standard  for  network  intercon¬ 
nection:  For  this  case,  it  is  assumed  that  every  network  offers 
a  common  datagram  service.  A  uniform  address  space  makes  it 
possible  for  subscribers  on  any  network  to  send  packets  ad¬ 
dressed  to  any  other  subscriber  on  a  connected  network.  Pac¬ 
kets  are  routed  between  subscriber  and  gateway  and  between 
gateways  based  on  the  destination  address.  No  attempt  is 
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livery  lo  the  Jestination  Inilividual  Jaiatiiainv  may  he  freely 
routed  through  different  gateways  to  recover  from  failures  or 
to  allow  load-spliitirig  among  parallel  gateways  joining  a  pair 
ot  networks 

The  gateway  gateway  interface  may  he  different  than  the 
network  access  interface,  if  need  he  (see  Fig.  ^1. 

This  strategy  requires  that  all  networks  implement  a  com¬ 
mon  interface  for  subsenhers.  The  simplicity  and  flexibility 
of  the  datagram  interface  strategy  is  offset  somewhat  by  the 
need  for  all  networks  to  implement  the  same  interface.  This  is 
true  for  the  pure  virtual  circuit  interface  strategy  as  well,  as 
will  be  shown  below 

One  of  the  problems  which  has  to  be  faced  with  any  net¬ 
work  interconnection  strategy  is  congestion  control  at  the 
gateways  If  a  gateway  finds  that  it  is  unable  to  forward  a 
datagram  into  the  next  netwoik,  it  must  have  a  way  of  reject¬ 
ing  It  and  quenching  the  flow  of  traffic  entering  the  gateway 
en  route  into  the  next  network  The  quenching  would  typi¬ 
cally  take  the  form  of  an  error  or  flow  control  signal  passing 
from  one  gateway  half  to  another  on  behalf  of  the  associated 
network  Similar  signals  could  be  passed  between  subsenbers 
and  the  packet  network  for  similar  reasons.  Since  datagram 
service  does  not  undertake  to  guarantee  end/end  reliability. 
It  IS  possible  to  relieve  momentary  congestion  by  discarding 
datagrams,  as  a  last  resort 

bl  I’irduil  circuits  for  network  interconnection  Another 
alternative  standard  network  service  which  could  be  used  for 
network  interconnection  is  virtual  circuit  service  (F'lg.  10). 
Independent  of  the  precise  interface  used  to  "set  up”  the 
virtual  circuit,  a  number  of  implementation  issues  immedi¬ 
ately  anse  if  such  a  service  is  used  as  a  basis  for  network 
interconnection 

Since  It  1$  intended  that  all  packets  on  a  virtual  circuit  be 
delivered  to  the  destination  subscriber  in  the  same  sequence 
as  they  were  entered  by  the  source  subscriber,  it  is  necessary 
that  either  1 )  all  packets  belonging  lo  the  same  virtual  circuit 
take  the  same  path  from  source  subscriber,  through  one  or 
more  gateways,  to  destination  subscriber,  or  ')  all  packets 
contain  sequence  numbers  which  arc  preserved  end-to-end 
between  the  source  DCF'  m  the  originating  network  and  the 
destination  DCF  in  the  terminating  network 

In  the  first  case,  virtual  circuits  are  set  up  and  anchored  to 
specific  gateways  so  that  the  sequencing  of  the  virtual  circuit 
service  of  each  network  can  be  used  to  preserve  the  packet 
sequence  on  delivery  This  results  m  the  concatenation  of  a 
senes  of  virtual  circuits  through  each  gateway  and,  therefore, 
the  knowledge  of  each  virtual  circuit  at  each  gateway  (since 
the  next  gateway  to  route  the  packet  through  must  be  fixed 
for  each  virtual  circuit). 

In  the  second  case,  there  is  no  need  to  restrict  the  choice  of 
gateway  routing  for  each  virtual  circuit  since  the  destination 
DCF  will  have  sufficient  information  to  resequence  incoming 
packets  prior  to  delivery  to  the  destination  subsenber. 

In  either  case,  the  destination  DCF  will  have  to  buffer  and 
resequence  packets  arriving  out  of  order  due  either  to  dis¬ 
ordering  within  the  last  network  or  to  alternate  routing  among 
networks,  if  this  is  permitted.  Some  networks  may  keep 
packets  in  sequence  as  they  transit  the  network.  I'his  wUI  only 
be  advantageous  at  the  destination  DCF  if  the  packets  enter 
the  network  m  the  desired  sequence.  If  such  a  service  is  relied 
upon  in  the  internet  environment,  then  each  gateway  must 
assure  that  on  entry  to  such  a  net.  the  packets  are  in  the  de- 
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l-lS  10.  Virtual  circuit  nctcoik  irilcrcoimccluMi  slfalegicv.  (a)  Sub 
scrihrr  baaed  gateway.  Iniernei  source  and  destination  carried  in  usei 
data  lieKl  of  .\.sS  call  set  up  packets,  (hi  \.7S  based  gateway.  Note 
how  much  of  the  X.25  Vt'  service  is  lerminaied  at  the  STh.  (c)  X.7S 
baaed  gateways  with  general  virtual  circuit  networks. 


Sired  order  for  delivery  to  a  destination  subscrifier  or  another 
gateway. 

The  huffenng  and  resequencing  of  packets  within  the  net¬ 
works  or  at  gateways  introduces  substantial  variation  in  buffer 
space  requirements,  packet  transit  delays,  and  the  potential  for 
buffer  lockups  lo  occur  1 50) .  (.'11.  (61 1 

If  packets  for  a  specific  virtual  circuit  are  restricted  to  pass 
through  a  fixed  senes  of  gateways,  and  if  a  standard  flow- 
control  method  is  agreed  upon  as  part  of  the  virtual  circuit 
.service,  then  it  is  possible  for  each  internet  gateway  to  partici¬ 
pate  m  end-lo-cnd  flow  control  by  modifying  the  flow  control 
information  carried  in  packets  carried  end-to-end  from  the 
sciurce  DCF  to  the  destination  DCF.  Consequently,  a  gate 
way  may  be  able  to  adjust  the  amount  of  traffic  passing 
through  it  and  thereby  achieve  a  kind  of  internet  gateway 
congestion  control.  If  this  is  done  by  alUx'ating  buffer  space 
for  “outstanding”  packets,  then  either  the  gateways  must 
guarantee  the  advertised  buffer  space  or  there  must  be  a  re¬ 
transmission  capability  built  into  the  internet  virtual  circuit 
implementation,  perhaps  between  source  DCF  and  destination 
DCF  or  between  DCF's  and  gateways. 

Such  a  mechanism  Joes  not,  however,  solve  the  problem  of 
network  congestion  unless  the  gateway-flow  control  decisions 
take  into  account  resources  both  in  the  gateway  and  in  the 
rest  of  the  network.  .Although  it  is  tempting  to  assume  that 
virtual  circuit-flow  control  can  achieve  internetwork  conges 
tion  control,  this  is  by  no  means  clear,  and  is  still  the  subject 
of  considerable  research. 

.As  a  general  rule,  compared  to  the  datagram  method,  the 
virtual  circuit  approach  requires  more  state  information  in 
each  gateway,  since  knowledge  of  each  virtual  circuit  must  bc 
maintained  along  with  flow  control  and  routing  information 
The  usual  virtual  circuit  interface  is  somewhat  more  complex 
for  subscribers  lo  implement  as  well,  because  of  the  anuvunt 
of  Slate  information  which  must  be  shared  by  the  siibscriK': 
and  the  local  DCF.  For  example,  implementations  of  the  \.d.^ 
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teijuire  4000-SS100  words  of  memory  on  Digital  l  iiuipmeni 
S  orporalion  f’Uf’  I  1  eomputers  By  eonlrast.  the  ARPANl  1' 
ind  Paekcl  Radio  Vetwotk  dalattram  interfaces  require  SlUV 
lOOO  words  ol  memory  on  the  same  machine,  f-'or  internet¬ 
work  operation,  this  may  be  even  more  burdensome,  since 
any  failure  at  a  gateway  may  require  a  subscriber-level  re¬ 
covery  throuKh  an  end-to-end  protocol,  in  addition  to  the 
virtual  circuit  interface  software,  as  is  shown  in  15-1 

Nevertheless,  it  may  be  advantajieous  to  consider  internet¬ 
working:  standards  which  usefully  employ  both  datagram  and 
virtual  Circuit  interlaces  and  services  I'or  example,  some 
spei  lal  internet  services  such  as  iiiullidcslination  delivery  may 
be  more  efficient  if  they  ate  first  set  up  by  control  exchanitcs 
between  the  subscriber  and  the  local  network  and  perhaps 
tialcw.iys  as  well  Once  set  up.  however,  a  datatirani  iiuvde  of 
operation  may  be  fai  mote  efficient  than  maintaining  virtual 
Circuits  fe't  all  deslinatic'ns  Implicit  virtual  circuits  which  are 
activated  by  simple  datagram-like  interfaces  are  also  attrac¬ 
tive  for  very  simple  kinds  of  terminal  equipment. 

If  It  IS  not  possible  for  all  networks  to  implement  a  common 
network  access  inlertace.  then  the  next  oppoitiinily  is  to 
slandardue  only  the  objects  which  pass  from  one  net  to  the 
next  and  to  mininii/e  any  requirements  for  the  sequencing 
of  these  objects  as  they  move  from  net  to  net 

ilenfra!  Hint  tiateways.  In  this  model,  a  gateway  is 
indistinguishable  from  any  other  netwevrk  host  and  will  imple¬ 
ment  whatever  host/network  interface  is  required  by  the 
networks  to  which  it  is  attached  For  many  networks,  this 
may  K’  \  25.  but  the  strategy  does  not  rely  on  this.  The 
principle  assumpticm  is  that  packet  networks  are  at  least 
capable  of  carrying  subscriber  packets  up  to  some  maximum 


length,  which  may  vary  from  network  to  network  It  is  specif¬ 
ically  not  assumed  that  these  packets  will  be  delivered  in  order 
through  inlermedi.ite  iietwoiks  and  gateways  to  the  destina¬ 
tion  host  I'his  minimal  type  of  service  is  often  termed  “data- 
gram"  service  to  distinguish  it  from  sequenced  virtual  circuit 
service  A  detailed  discussion  of  the  tradeoff  between  data¬ 
gram  and  virtual  circuit  types  of  networks  is  given  elsewhere 
1521 

The  basic  model  of  network  interconnection  for  the  data¬ 
gram  host  gateway  is  that  internetwork  datagrams  will  be 
earned  to  and  from  hosts  and  gateways  and  between  gateways 
by  encapsulation  of  the  datagrams  in  local  network  packets 
l\vu/in  deiicnbes  this  process  genencally  as  "wrapping"  [371 
rile  basic  internetwcuk  service  is  therefore  a  datagram  ser¬ 
vice  rather  than  a  virtual  circuit  service  The  concept  is 
illustrated  in  Kig  1  1 

Datagiam  service  does  not  offer  the  subscriK'r  as  many 
facilities  as  virtual  circuit  service.  For  example,  not  all  data¬ 
grams  are  guaranteed  to  be  delivered,  nor  do  those  that  are 
delivered  have  to  be  delivered  in  the  -wquence  they  were  sent 
Virtual  circuits,  on  the  other  hand,  do  attempt  to  deliver  all 
packets  enterevl  by  the  source  in  sequence  tvv  the  destination 
I'he.se  relaxations  allow  dynamic  routing  of  datagrams  among 
multiple,  internetwork  gateways  without  the  need  for  sub¬ 
scriber  intervention  or  alert 

The  internet  datagram  concept  gives  subsenbers  access  to  a 
basic  internet  datagram  service  while  allowing  them  to  build 
more  elaborate  end-to-end  protocols  on  top  of  it  Fig.  1  2 
illustrates  a  po-ssible  protiKol  hierarchy  which  could  be  based 
on  the  internet  datagram  concept  The  basic  internet  data¬ 
gram  service  could  be  used  to  support  transaction  protcvcols 
or  real-time  protocols  (RUM  such  as  packet-voice  protocols 
(PVl’l  which  do  not  require  guaranteed  ot  sequenced  data 
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Fig.  1 2.  Protocol  layering  with  internetwork  datagrame. 


NET  A  ENDiEND  SUBSCRIBER  PROTOCOL  NET  B  ENO/ENO  SUBSCRIBER  PROTOCOL 


delivery;  reliable,  sequenced  protocols  could  be  constructed 
above  the  basic  internet  datagram  service  to  perform  end/end 
sequencing  and  error  handling.  Applications  such  as  virtual 
terminal  protocols  (VTP)  140),  (42),  (48)  or  file-transfer 
protocols  (40),  (42),  (49)  could  be  built  above  a  reliable, 
point-to-point,  end/end  service  which  is  itself  built  atop  inter¬ 
net  datagrams.  Under  this  strategy,  the  basic  gateway  func¬ 
tions  ate  the  encapsulation  and  decapsulation  of  datagrams, 
mapping  of  internet  source/destination  addresses  into  local 
network  addresses  and  datagram  routing.  Gateways  need  not 
have  any  knowledge  of  higher  level  protocols  if  it  is  assumed 
that  protocols  above  the  internet  datagram  layer  are  held  in 
common  by  the  communicating  hosts.  Datagrams  can  be 
routed  freely  among  gateways  and  can  be  delivered  out  of 
sequence  to  the  destination  host. 

The  basic  advantage  of  this  strategy  is  that  almost  any  sort 
of  network  can  participate,  whether  its  internal  operation  is 
datagram  or  virtual  circuit  oriented.  Furthermore,  the  strategy 


offers  an  easy  way  for  new  networks  to  be  made  “backwards 
compatible,”  with  older  ones  while  allowing  the  new  one?  to 
employ  new  internal  operations  which  are  innovative  or  more 
efficient. 

Every  subscriber  must  implement  the  internet  datagram  con¬ 
cept  for  this  strategy  to  work,  of  course.  The  same  problem 
arises  with  the  standard  network  interface  strategy  since  all 
subscribers  must  implement  the  same  network  interface. 

4)  Protocol  Translation  Gateways:  It  would  be  misleading 
lo  claim  that  the  concept  of  protocol  translation  has  not 
played  a  role  in  the  discussion  thus  far.  In  a  sense,  the  encap¬ 
sulation  of  internet  datagrams  in  the  packet  format  of  each 
intermediate  network  is  a  form  of  protocol  translation.  The 
basic  packet  carrying  service  of  one  network  is  being  trans¬ 
lated  into  the  next  network's  packet  carrying  service  (see  Fig. 
13).  This  concept  could  be  extended  further.  For  example, 
if  two  networks  have  a  virtual  circuit  concept,  one  imple¬ 
mented  within  the  subnetwork  and  the  other  through  common 
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hi>.st;lu)M  j>r(it>H  uls.  It  mijthi  he  pusMblc,  j|  the  t(>>tcwjy  tic 
tween  the  nets,  In  m,ii)  one  network  s  virtual  circuit  into  ttie 
other's  This  same  idea  coiiKl  he  applied  to  higher  level  proto¬ 
col  inappinits  as  well,  for  instance,  the  virtual  terminal  proto¬ 
col  tor  one  network  might  he  transtormed  into  that  ol  another 
"on  the  n>  " 

Vhc  success  ol  such  a  translation  strategy  depends  in  large- 
part  on  the  commonality  of  concept  hetween  the  protocols 
to  he  translated  Mismatches  in  concept  may  reiimre  that  the 
service  obtained  m  the  concatenated  case  he  a  subset  of  the 
services  obtainable  fusm  either  of  the  tsvo  services  being  trans 
lated  I’stending  such  lianslations  through  several  gateways 
can  be  difficult,  particularly  if  the  protocols  being  tran.sliited 
do  not  share  acommon  address  space  for  internetwoik  sources/ 
destinations  In  the  estremc,  this  strategy  can  result  in  sub- 
scriheis  "logging  in"  to  the  gateway  in  order  to  activate  the 
protocols  of  the  next  nctwoik  Indeed,  front-end  computers 
could  be  cousideied  degenerate  translaliisu  gateways  since  they 
transform  host  front  end  protocols  into  network  protocols 

There  aie  circumstances  when  lianslation  cannot  be  avoided 
I'oi  instance,  when  tlie  protocols  of  one  network  cannot  be 
modified,  but  internet  seivice  is  desired,  there  may  tic  no 
alternative  but  to  implement  protocol  translations  The  model 
typically  used  to  guide  protocol  lianslation  gateways  is  that 
the  source  destination  hosts  lie  on  either  side  of  the  ttansla- 
lion  gateway  ('oncalenalion  of  protocol  translations  through 
several  networks  and  gateways  is  conceivable,  but  may  be  very 
ditficult  in  practice  and  may  produce  very  inefficient  service 

('  Wntii's,  l</i/ri'i.ii  J,  urn/ A’liiiti-.i 

In  \'tdet  lo  manage,  contiol,  and  suppoit  coinmiinic,ition 
,iinong  computeis  on  one  oi  more  nelwoiks,  it  is  es,senlial  Unit 
conventions  be  esiablistied  tor  identitying  Ihe  communicators 
Tor  purposes  ot  this  discussion,  we  will  use  llie  leiin  hosl  lo 
reter  lo  all  conipiilers  wIikIi  atlacli  lo  a  nelwoik  at  llie  net 
work  access  level  of  protocol  (see  Table  I)  Subscribers  lo 
terminal-access  servieescan  be  thought  of  as  attaching  to  hosts, 
even  if  Ihe  liosi  is  embetided  in  lire  hardware  and  soflwaieof 
a  packet  switch  as  a  layer  ol  protocol  t'onse>|iienlly,  we  lan 
say  that  the  basic  task  of  a  packet-switching  network  is  to 
transport  data  fioiii  a  source  liosi  lo  one  or  more  deslinalion 
liosis 

To  acconiplisli  this  task,  eacli  nelwoik  needs  to  know  to 
whicli  destination  packets  arc  lo  be  delivered  Tven  in  broad 
cast  nets  such  as  Ihe  TTIITKNTT,  this  information  is  neces¬ 
sary  so  that  Ihe  desliiialioii  host  laii  discriminate  packets 
destined  fot  itscll  from  all  otheis  lieard  on  the  net  At  the 
lowest  protocol  levels  it  is  typical  to  associate  destinations 
with  aJJrfSics  An  addiess  may  be  simply  an  integer  or  it 
may  have  moie  internal  structure 

At  liighei  levels  of  protocol,  howevei,  it  is  moie  common  to 
Imd  text  strings  sucli  as  "Mill  TU'S"  or  "BUN- TTNTX"  used 
,is  ruinii-3  of  destmatuins  Application  software,  siicli  as  elec- 
lionn  mail  services,  iiiiglil  employ  such  names  along  with 
more  refin- d  destination  identifiers.  Tot  example,  one  of  Ihe 
authors  has  an  electronic  mailbox  named  "KIKSTTIN  at  ISI" 
located  in  a  computer  at  the  University  of  Southern  California's 
Information  Sciences  Institute. 

Typically,  application  programs  tiansform  names  into  ad 
dresses  which  can  be  understood  by  the  packet-switching  net 
work.  The  networks  must  transform  these  addres.ses  into 
riiutes  lo  guide  Ihe  packets  lo  their  destination.  Some  net 
works  bind  addres.ses  to  routes  m  a  relatively  rigid  way  feat.. 


selling  uji  viitiiaj  ^iKuiis  will.  Iixcd  roulingi  while  olhen 
dcicinuiie  routes  as  llie  packets  move  tiom  swiidi  lo  wiicb 
c  hoosmg  alternate  loutcs  ui  by  pace  laded  oi  c  ongesteu  aieas  c'l 
the  network  Broadcast  nelwoiks  need  not  create  r.nilc' at  all 
le  g  .  SATM  T1 

In  simple  terms,  a  twn,  tells  what  an  .'bied  is.  an  j./dri  n 
tells  where  it  is,  and  a  'our.-  tells  how  ic<  gel  Ihctc  iS-t|  \ 
simple  model  mvolsmg  these  three  concepts  is  that  hosts  nans 
loini  names  inli<  addiesses  amt  nelwoiks  li.insloim  addresses 
into  routes  Ilf  necessaiv  )  lloweier.  this  basic  model  doesleave 
a  laige  mimbet  of  loose  ends  1  he  siibieci  is  sc>  filled  with 
issues  that  it  is  not  possible  in  this  pa|ier  to  explore  them  all  m 
depth.  In  what  Krllows.  vune  ot  Ihe  major  issues  are  raised 
and  some  partial  lesolulions  aie  ofteied. 

I'lie  maioi  ctiiestion  is  "Which  objects  m  the  nelwoik  should 
have  names'’  addre.sses'’"  I’oii/m  and  /immeimann  olfei  a 
number  ol  views  on  this  niieslion  in  their  paper  in  this  issue 
1  t7|  A  generic  answer  might  be  llial  at  least  all  objects  which 
can  I'c  addiessed  try  the  network  should  have  names  as  well  so 
that  high  level  protocols  can  reler  to  them  Toi  example,  it 
might  be  reasonable  for  every  hosl  connection  on  Ihe  network 
lo  have  an  name  and  an  address.  There  also  may  be  objects 
mieriial  li)  Ihe  nelwoik  which  also  have  addresses  siu  h  as  the 
slalislics-galhering  JjKc  hi>sls  m  Ihe  AKI’AM-  I  |  Ik  1 

A  related  issue  is  whelhei  objects  should  ot  can  have  inultiple 
names,  multiple  addresses,  and  multiple  routes  by  which  they 
can  be  reached  The  most  general  n.’soliilion  ol  this  issue  is  lo 
I'eimit  multiple  names,  addresses,  and  routes  to  exist  lot  the 
same  object  An  example  taken  fiom  the  miillmetwork  en- 
viioiimenl  may  seive  lo  illusliale  this  notion  Tig.  (<  shows 
Ihiee  networks  which  ,i;e  mlcrionnecled  by  a  mimbei  >'1  gate¬ 
ways  Tach  gateway  loi  pan  ol  gateway  halvesl  has  two  inter 
fa.es,  one  lo  each  nclw.nk  lo  which  it  is  allachcd  Tlainly 
Ihcie  IS  the  possibility  lli.il  scvci.il  alternate  routes  pas.sing 
Ibioiigh  diflerenl  gateways  and  nelwoiks  could  be  used  lo 
cany  packets  from  a  soiiicc  hosl  m  one  tut  lo  a  destination 
hosl  III  allot hei  net  This  is  msl  Ihe  analog  ol  alleinale  loiilmg 
within  a  single  netw.nk 

T'lirlliei more,  each  gateway  has  Iw.i  addiesses.  lyjiically  one 
lor  each  allached  nelwoik  Tins  is  just  Ihe  anaU'g  ol  a  hosl  on 
one  network  allached  l.i  Iwo.n  more  packet  swilctieslor  reh 
ability  The  term  mulliluinung  is  ollen  used  to  icier  to  mul¬ 
tiply  attached  hosts 

T'mally,  it  may  be  iisi-lul  lo  penuil  a  gateway  l.i  have  more 
than  one  name,  foi  example,  one  foi  each  nelwenk  to  which  it 
IS  attaclied  This  might  allow  high-level  piolocols  to  force 
I'ackcls  1.1  he  routed  m  ceilain  ways  for  diagnostic  oi  iithei 
teasons  Multiple  naming  also  allows  Ihe  use  ot  nicknames  foi 
usei  convenience  Manv  of  these  same  comments  would  apply 
lo  hosts  attached  l.i  miilliple  networks 

An  interesting  addiessmg  and  routing  problem  arises  in  mo¬ 
bile  packet  radio  networks  Snue  li.isls  are  tree  lo  move  about, 
Ihe  uetwciik  will  need  to  dynamic  ally  change  the  routes  used  lo 
reach  each  hosl  l  or  robustness,  il  ts  also  desirable  that  hosts 
be  able  lo  attach  dvnainically  lo  ditferi-nl  packet  radios  Thus 
failure  of  a  pac  ket  radio  need  not  prevent  hosts  from  accessing 
llie  network,  t  his  iei|uires  that  hosl  names  and  perhaps  host 
addres,ses  be  decoupled  lioin  packet  radio  a.Idresses  The  net 
work  iniisl  be  able  to  search  fot  hosts  or  alleinatively .  hosts 
must  "report-m"  to  Ihe  network  so  that  their  addres.srs  can  be 
associated  with  the  attached  packet  radio  to  laeililate  route 
selection  based  on  host  address  This  is  just  a  way  of  su|ipoil 
mg  hi>sl  iii/>/re.t,wnx'  lather  than  using  Ihe  mote  eoinmon 
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Z'rti  jiio.  fti’O  III  wtiiili  d  tiiiM  v  dililtonx  is  an  cxioii 

stun  ot  the  I'dsket  switch  adJievs 

A  viiiudl  issue  III  iielNsoik  iiileio'iineetion  is  the  esieiil  In 
whuh  it  shoiiM  oi  must  iiiipiul  adtliessinii  proo’iUiies  whuli 
ate  uliosy lu ralu  in  a  lurtioilar  nelssiitk  It  is  silearilaReniis 
mil  In  ie(|uiie  the  suhseiiheis  no  eat  h  netwnik  In  have  ilelailetl 
kiiowleilKe  nl  Ihr  nelwtuk  atliliess  .wruifurr  nl  all  iiileiinii 
nfsIeO  iielwnrks  One  pnssihililv  is  In  slaiulanli/e  an  inleiiiel 
wniK  aiUlievs  sliiieluie  whuh  >an  be  mapped  into  Ineal  net 
wnrK  adilresses  as  neeiled,  eilhei  by  subsenbcis,  by  Ralewavs 
ni  by  Knb  Subsiiibeis  wnubl  knnw  hnw  In  map  inleinelssnik 
serviee  names  inin  addresses  til  the  Inim  Nl- 1  WtlRk/SI- RVI  R 
Subsenbeis  need  nnl  Wnnss  the  line  sinuture  nl  the  SI-RVI-R 
field  Oaleways  wniild  mute  paekets  nn  the  basis  nl  the  M- 1 
WeIRK  pail  nf  the  atidiess  unlil  reathmtl  a  Ralevsay  allatlied 
In  the  netssiiik  ideiililied  by  Nl  IWORK  At  this  pninl.  the 
liateisas  mniht  inteipiel  the  SI  RV  I  R  pail  nf  the  addiess,  as 
iieeessaiy,  In  eause  the  paeket  l»i  be  deliveied  In  the  desiied 
ht'sl 

The  addievsiiiR  slialeRV  piesenlly  iiiulei  ennsideralinn  by 
f'Ciri  l\  121,  l.Ull  I  IS  based  nn  the  lelcphniie  netwnik  |lp 
m  14  diRits  ean  be  usi'tl  in  an  addiess  I'he  lust  4  diRilsaie  a 
"destinalinn  nelwnik  iden'ilieatinii  ende"  vu  PNK'  Snme 
eouniries  are  allnealeil  iiinie  than  niie  l>NU'  |lhe  Unileil  Slates 
has  200)  The  lemaininii  ten  diRils  may  be  used  in  iinpleinenl 
a  hieiarehieal  addiessinii  sinuliite,  mueh  like  the  tine  used  in 
the  exislum  leleplnme  nelwnik 

Sinee  the  ('('II  I  aRieeiiienls  aie  Ini  uileiiialinnal  npeialinii. 
It  miiihl  be  fair  Ui  as.suine  that  the  United  States  will  nnl  need 
more  than  200  publn-  nelwtuk  idenlifieis.  Ilnwevei,  this 
seheme  dnes  nnl  take  iiiln  aestninl  the  need  fni  addiessini; 
private  iielwntks  I'he  piivale  iietwniks,  uiidei  this  addrevsmi! 
ptneediiie  will  iimsl  likely  appear  Iti  be  a  enlleelniii  nl  nne  t>i 
innie  terminals  ni  linst  enmpiiteis  nn  nne  nr  mnie  piiblie  net 
svnrks  It  is  Inn  early  In  lell  linw  niiuli  this  asy  ninieliy  m  ad 
diessiii)!  between  piiblie  and  piivale  netwniks  will  atleet  piisate 
multinetwnrk  prntnenis 

\  related  pmblem  whuli  is  lU'l  uiiu|ne  tt>  netwnik  inleisnn 
iieelinii  has  In  dn  willi  addievsiiiR  (really  mult i|i|exui|i  and  tie 
multiplexinii)  at  liigliei  ptnlnenl  levels  Hie  publu  sariiers 
lend  In  nffei  servues  fni  tei  initial  as  well  as  Imsl  aeeess  In  net 
wiirk  faeilities  Mils  lypually  means  that  adiliesses  must  be 
asMtined  lii  terminals  I'he  is.siie  iswhelliei  the  leiniitial  addiess 
shniild  be  assneialed  with  nr  independent  nl  the  pininei'is 
used  In  siii'pnil  teiminal  tn-linsl  enmiiiiinualinn 

The  preseiil  iiiiniberinii  svlieme  wniiUI  nnt  distinitiiisli  be 
tween  a  hnsi  addiess  aitil  a  terminal  adilre.ss  A  Imsl  itliKhI 
have  many  adtlresses,  each  enirespnndiiiii  tti  a  piiieess  wailiiiR 
In  serviee  ealliiiit  termiiials. 

I  here  has  been  diseiissinn  within  ('CM  T  I'niieeininti  "siibad 
dtessiii)!"  Ilitnuiili  Ihe  use  nf  a  usm  data  field  earned  in  viiliial 
eall  "setup"  paekets  This  nnlion  wnuld  support  Ihe  enneepi 
nl  a  sinjtle  Imsl  address  with  leriiiimil  nr  prneess  level  deniiilli 
plexuiK  aehieved  thrniigh  Ihe  use  nf  the  user  data  field  sub 
addre.vxinit 

It  seems  reasiiiiable  to  prediel  that,  as  lerminals  ineiease  in 
eninplexity  and  eapability.  it  will  eventually  be  attraetive  In 
support  inulliple  e-oiieurrent  assneialinns  bet  ween  the  lerminal 
and  several  remnic  serviee  taeililies.  Appheatinns  rei)uiiin|t 
this  eapabilily  will  need  terniiiial  multiplexinii  ixuivenlinns 
beyond  those  eurrently  provided  for  in  Ihe  CCI  V  V  reenmmen 
dalinns. 


In  simplily  implemenlatu'ns  nf  inteinel  pinlnml  snflwaie. 
If  IS  essential  In  plaee  bniinds  nn  the  nuximuni  si/e  nl  Ihe 
Nl  I'WORK 'SI  RVI  R  addiess  Olheiwise,  subsenbeis  may 
have  In  eniisliiiel  name  In  address  mappinil  tables  with  arbi 
Irartly  laiitr  and  mmplex  entries 
I'ven  il  all  these  issues  aie  revnlved,  there  is  still  a  qiieslinn  nl 
"sniliee  iniitiiiit  "  in  whuli  a  siibseribei  deliiies  Ihe  mute  In  be 
taken  by  a  pailuiilai  paeket  nr  viitual  siisiiil  DependinR  s'li 
(he  laiiRe  nf  infeitielwnik  serviees  available,  a  siibasrthet  may 
want  Is'  eniKinl  I'aekel  mutes  II  is  nnt  yet  eleai  how-  sush  a 
eapabilily  will  uileiast  with  assess  siiiilml  eisnvenluins,  but 
this  may  be  a  desirable  eapabilily  if  (lateways  are  nnl  able  In 
aiils'iiiatieally  seleet  mutes  whieh  maleh  user  servis-e  res^uire 
iiieiils 

/•  /'low  tin, I  I 'nnxvvnnn  ('nnlrn/ 

I  s'i  puipnses  nl  divussinn,  we  distinguish  between  flow  and 
enngestiisn  ennlrnl  l-'hiw  i-nnimi  is  a  ptni-edure  Ihmiiith  whieli 
a  pair  nf  ennitniinu  ainrs  ie)iula(e  tiaOie  flnwiiiR  fisim  anuree 
tn  destinalinn  (oas'li  diieetu'ii  piissibly  beiiiii  dealt  with  sepa 
lately)  ('niiiteslinn  esiiilml  is  a  pmeediiie  wlieieby  disliibiiled 
nelwiirk  lessiuis-es,  siieli  as  ehannel  bandwidth,  buffer  eapaeily . 
CI’U  eapaeity,  and  the  like  ate  pmleeted  from  isvemubserip 
Ill'll  by  all  sniiiees  nf  net  work  Iraffie  In  Reneial.  Ihe  siiei'evs 
fill  ivperatmn  nf  flsiw  enntinl  pmeediires  fs't  every  pan  s'l  iiei 
work  snininuiiieanls  ds'es  nnl  ttiiaiantee  that  the  nelwsuk 
less'iirees  w  ill  remain  uns's'iiiiesled 
In  a  siiiRle  iielwsirk,  Ihe  I'niilrnl  iif  fliiw  and  si'iiitesfinii  is  a 
'■simplex  and  lu't  well  uiisleislsii'd  pmblem  In  a  iiiiiKiiiel work 
envimnmenl  it  is  even  nmie  snmplex.  nwiiiR  ts'  Ihe  possible 
vaiialiniis  in  Ibiss  aiul  s-nniieslinn  si'iilml  pnlieies  fnunsi  in 
eaeh  ei'iisliluent  iielws'ik  l-ni  example,  si'iiie  nelws'iks  may 
ii)!islly  ei'iilml  (he  iiipiil  s>(  paskets  iiili'  the  netwntk  and  ex 
pliiitlv  lule  nut  dmppinii  paekets  as  a  means  s'f  s-s'njestK'ii 
eniitii’l  At  the  I'lhei  extieme,  snme  nelwsiiks  may  drop 
paekets  .is  Ihe  si'le  means  nl  eiiiiiteslu'n  es'iilm) 

,Al  this  slaRe  ssf  develsipment,  very  little  is  know n  abniil  the 
I'ehavinr  "1  ennsestinn  in  niiilliply  interenniieeled  iietw'vrks 
It  IS  eleat  that  s.'iiie  iiiei  hanisms  will  be  ies)iiued  whieh  perniil 
gateways  and  nelss>'iks  tn  asss-it  es'iitml  I'vei  liaflie  influx  es 
peeially  when  a  (talessay  'niineets  nelwsuks  I'f  widely  varyitlti 
eapaeily  Miis  pmblem  is  likely  In  be  iimst  visible  at  (lateways 
ininin)!  liijth  speed  Ineal  netwniks  In  Kiri)!  haul  I'liblie  nets 
rile  peak  rales  nl  the  loial  nets  mnilil  exeeed  that  nl  the  Innii 
haul  nets  b\  taeti'is  i<t  .10  1 00  iii  nu're  (!eiieii'  pmeediires 
aix'  needed  h'l  n.iteway/iielw'iik  and  iialeway  (taleway  flow 
and  enii|iestinn  ennlii'l  ,Siieh  pmbleins  also  show  ii|’ in  .xiiiftle 
networks,  but  are  amplified  in  the  iiiiiltinelwnik  easi' 

I-  luniinrmy 

.AeeniitifiiiR  ti'i  iiitenietwnrk  (taffie  is  an  iiiipi'itaiil  pn'b 
leiii  I  he  piiblie  netwniks  need  nieehanisms  for  revenue  shai 
in(!  and  subs' ribeis  need  simple  pmeeduies  fiu  verifymR  Ihe 
ai  eiiraev  I'f  nelwiuk  provided  aeeniinlmiis 
I'he  piiblie  paeket  swiiehinit  networks  appear  to  be  ennverg 
ing  on  pmeediires  whieh  aeeoiint  for  siibseiibei  use  on  Ihr 
basis  I'f  the  minibei  of  viitiial  eireiiils  eiealed  dining  Ihe  ae 
eoiiiiling  peiind  ami  the  niuiibri  of  paekets  sent  oneat-h  viitiial 
eireiiit  Indeed,  it  has  been  argued  that  aeeoiintmg  I'n  the 
basis  I't  virtual  eireiiils  at  gateways  teiiuites  less  nvrthead  than 
aee'iunting  on  a  piiie  datagram  basis  1,121  Seenarios  van  be 
vileil  whieh  support  Ihe  opposite  'X'nelusioii 
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Suppose  there  is  a  choice  between  setting  up  virtual  circuits 
for  each  transaction  and  sending  a  datagram  for  each  transac¬ 
tion,  and  that  virtual  circuit  accounting  includes  information 
on  each  virtual  circuit  setup  (as  in  the  present  telephone  net¬ 
work).  If  datagram  accounting  simply  accumulates  the  number 
of  datagrams  sent  between  particular  sources  and  destinations 
without  regard  to  the  time  at  which  they  are  sent,  then  the 
amount  of  accounting  information  which  is  collected  for  the 
datagram  case  will  be  substantially  less  than  for  the  virtual 
circuit  case.  In  the  limit  (i.e.,  one  packet  per  transaction),  the 
virtual  circuit  accounting  information  is  proportional  to  2N, 
where  N  is  the  number  of  transactions,  while  for  the  datagram 
case,  it  is  proportional  to  log  A' (base  2).  This  is  simply  because 
the  datagram  case  only  sums  counts  for  traffic  between  source/ 
destination  pairs  while  the  virtual  circuit  accounting  would 
identify  start/stop  times  for  each  virtual  circuit. 

Alternatively,  if  the  bulk  of  the  traffic  involves  a  large  num¬ 
ber  of  packets  per  transaction,  then  the  two  accounting  pro¬ 
cedures  would  accumulate  more  nearly  the  same  information 
since  each  would  predominantly  involve  accounting  for  packet 
flow. 

If  it  is  chosen  not  to  account  for  virtual  circuit  duration,  but 
merely  to  account  independently  for  the  number  of  virtual 
circuits  and  the  number  of  packets  sent  between  source/desti¬ 
nation  pairs,  then  the  virtual  circuit  accounting  would  be  closer 
to  the  datagram  case. 

The  important  conclusion  to  be  drawn  is  that  accounting  for 
datagrams  is  generally  less  complex  than  accounting  for  virtual 
cucuits,  but  that  the  two  can  be  made  arbitrarily  similar  by 
suitable  choice  of  the  details  of  the  accounting  information 
collected. 

F.  Access  Control 

In  multinetwork  environments,  it  may  be  necessary  for  each 
network  to  establish  and  enforce  a  policy  for  “out-of-network” 
routing.  For  example,  a  public  network  might  conclude  agree¬ 
ments  with  other  networks  regarding  the  type  and  quantity  of 
traffic  it  will  forward  into  other  networks.  This  might  even  be 
a  function  of  the  time-of-day.  Consequently,  mechanisms  arc 
needed  which  will  permit  networks  to  prevent  traffic  from 
entering  or  leaving  or  to  meter  the  type  and  rate  of  traffic 
passing  into  or  out  of  the  network. 

Another  example  of  the  need  for  control  arises  with  the  pos¬ 
sibility  of  third-party  routing.  That  is,  traffic  destined  from 
network  A  to  network  B  is  routed  through  network  C.  It  can¬ 
not  be  assumed  that  all  networks  have  gateways  to  all  others. 
However,  some  nets  may  want  to  limit  the  amount  of  transit 
traffic  they  carry.  There  may  be  explicit  agreements  among  a 
subset  of  the  nets  regarding  revenue  sharing  for  transit  services. 
If  a  particular  network  does  not  have  a  revenue-sharing  agree¬ 
ment  with  the  particular  source/destination  networks  of  a 
given  virtual  circuit  or  datagram,  then  it  must  be  able  to  reject 
the  offending  traffic  if  it  so  chooses. 

There  does  not  seem  to  be  any  technical  barrier  to  separating 
the  access  control  policy  decision  mechanism  from  the  enforce¬ 
ment  of  the  policy.  For  example,  a  gateway  might  simply  en¬ 
force  policy  by  sending  traffic  for  which  it  has  no  known  ac¬ 
cess  rules  to  an  access  controller.  If  we  adhere  to  the  model 
that  gateways  have  two  halves,  then  each  half  deals  with  the 
network  to  which  it  is  connected.  The  iccess  controller  can 
either  dynamically  enable  the  flow  by  causing  table  entnes  at 
the  gateways  which  permit  the  flow  to  be  crested  or  it  can  tell 
the  gateway  to  reject  all  further  traffic  of  tl.it  type. 


Clearly,  access  control  policies  will  affect  routing  strategies, 
so  this  adds  a  compbcating  factor  into  any  internetwork  rout¬ 
ing  strategy  implemented  by  the  gateways.  At  present,  very 
little  experience  has  been  accumulated  with  internet  access 
control  and  routing  policies.  For  the  most  part,  agreements 
among  public  networks  have  been  bilateral  and  transit  routing 
has  been  treated  as  a  very  special  case.  When  EURONET  (6| 
becomes  operational,  this  problem  will  be  particularly  impor¬ 
tant  to  solve. 

G.  Internet  Services 

It  is  by  no  means  clear  what  set  of  services  should  be  stan¬ 
dardized  and  available  from,  at  least,  all  public  data  networks. 
The  current  CCITT  recommendations  provide  for  virtual  cir¬ 
cuit  service  and  terminal  access  service  on  all  public  packet¬ 
switching  networks. 

Although  the  recommendations  (X.3,  X.25)  provide  for  frag¬ 
mentation  of  packets  being  delivered  to  a  subscriber  on  a  vir¬ 
tual  circuit,  the  current  X.7S  gateway  draft  recommendation 
uses  an  agreed  maximum  packet  size  of  1 28  octets  of  data,  not 
including  the  header.  This  agreement  avoids  for  the  moment 
the  need  to  fragment  packets  crossing  a  network  boundary,  as 
long  as  all  subscribers  recognize  that  the  maximum  length  in¬ 
ternetwork  packet  allowed  is  128  octets.  Bilateral  exceptions 
to  this  rule  may  develop  but  neither  a  fixed  size  nor  a  collec¬ 
tion  of  special  cases  represent  a  very  general  solution  to  this 
problem. 

It  has  been  argued  |2S|  that  a  general  scheme  for  dealing 
with  fragmentation  is  desirable  so  that  new  network  technol¬ 
ogies  supporting  larger  packet  sizes  can  be  easily  integrated 
into  the  multinetwork  environment. 

Apart  from  fragmentation,  there  are  a  set  of  special  services 
such  as  multidestination  addressing  and  broadcasting  which 
could  be  used  to  good  advantage  to  support  multinetwork  ap¬ 
plications  such  as  teleconferencing,  electronic  mail  distribution, 
distributed  file  systems,  and  real-time  data  collection.  Other 
services  such  as  low  delay,  high  reliability,  high  bandwidth, 
and  high  priority  are  also  candidates  for  standardization  at  the 
internet  level. 

As  in  the  case  of  access  control,  selection  of  such  services 
might  constrain  the  choice  of  packet  routing  to  networks 
capable  of  supporting  the  desired  services.  Once  again,  very 
little  experience  with  standard  internet  services  has  been  ac¬ 
cumulated  so  this  subject  is  still  a  topic  for  research.  For  the 
most  part,  terminal-to-host  services  have  been  successfully  of¬ 
fered  across  network  boundaries  using  nearly  all  of  the  net¬ 
work  interconnection  methods  described  in  this  paper.  It 
remains  to  be  seen  whether  more  complex  applications  can  be 
equally  well  supported. 

VII.  X.25/X.75-THE  CCITT  STRATEGY  EOR 
Network  Interconnection 

The  common  network  access  interface  concept  is  favored  by 
CCITT  for  network  interconnection.  In  the  CCITT  model  of 
packet  networking,  all  networks  offer  the  same  interface  to 
packet-mode  subscribers  and  this  is  called  X.25.  X.2S  is  a  vir¬ 
tual  circuit  interface  protocol.  However,  gateways  between 
networks  employ  an  interface  protocol  called  X.75  |33], 
which  is  much  like  X.2S  but  accommodates  special  network/ 
network  information  exchange,  such  as  routing  information, 
accounting  information,  and  so  on. 

Fig.  10(a)  illustrates  the  basic  network  interconnection 
strategy  proposed  by  CCITT.  To  appreciate  the  difference 
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hclwcen  this  strategy  and  the  "common  subnetwork"  strategy. 
It  IS  necessary  to  have  some  understanding  of  the  X.  25  packet 
network  interface.  X.2S  provides  a  virtual  cucuit  interface  for 
the  setup,  use.  and  termination  of  virtual  circuits  between 
subscnbers  of  the  networks.  X.25  provides  for  flow  control  of 
packets  per  virtual  circuit  flowing  into  or  out  of  the  network. 
Subscribers  may  set  up  switched  virtual  circuits  by  sending 
“call  request"  packets  into  the  network  and  receiving  "call 
confirmation”  packets  in  return.  The  standard  also  provides 
for  permanent  virtual  circuits. 

The  public  networks  plan  to  employ  X.2S  interfaces,  it  can 
therefore  be  assumed  (hat  source  and  destination  hosts  in  dif¬ 
ferent  networks  will  essentially  want  to  exchange  “call  request" 
and  “call  accepted"  packets  through  the  medution  of  one  or 
more  gateways.  This  strategy  could  result  in  a  series  of  virtual 
circuits  chaining  source  host  to  gateway,  gateway  to  gateway, 
and  gateway  to  destination  host,  alternately  an  end-to-end 
virtual  circuit  could  be  set  up  from  source  host  to  destination 
host,  with  the  gateways  acting  as  relays  without  any  special 
knowledge  of  the  virtual  circuits  passing  across  the  network 
boundary. 

The  principle  difference  between  the  X.25  inteiface  and 
X  75  interface  is  that  virtual  circuit  setup  and  clearing  packets 
are  passed  transparently  by  the  X.75  gateway  to  the  next  gate¬ 
way  or  destination.  For  reasons  which  are  described  below,  it 
IS  necessary  to  maintain  the  sequence  of  packets  belonging  to 
a  given  X.25  vutual  circuit  as  they  pass  through  a  gateway  and 
enter  the  next  network.  Therefore,  a  virtual  circuit  is  in  fact 
created  between  the  source  host  and  intermediate  gateway  and 
between  gateways.  The  X.75  gateway  does  not  spontaneously 
generate  any  “call  acceptance"  packets  in  response  to  "call 
request"  packets,  but  it  does  participate  in  (he  sequencing  and 
flow  control  of  packets  on  each  virtual  circuit  passing  through. 
Other  differences  between  the  X.25  and  X.75  interface  have  to 
do  with  the  nature  of  the  internetwork  accounting  or  routing 
information  which  might  be  exchanged  over  X.75  which  would 
not  be  appropriate  for  a  subscriber  to  exchange  with  the  net¬ 
work  over  the  X.25  interface. 

The  design  of  the  X.75  type  of  gateway  depends  in  principle 
upon  all  networks'  use  of  the  X.25  subscriber  interface.  Some 
networks,  like  the  FTMFRNiiT,  cannot  implement  it  without 
extensive  modification,  because  there  are  no  packet  switches 
in  the  network  to  support  the  required  packet  reordering  at 
the  destination.  The  alternative  is  to  insist  that  all  internet 
applications  rely  on  a  sequenced  data  protocol  built  into  the 
hosts  or  front-ends.  For  some  services,  such  as  packet  speech, 
the  potential  overhead  of  resequencing  packets  before  delivery 
to  the  destination  may  prevent  the  service  from  being  viable. 
This  problem  could  be  amplified  if  packets  are  constrained  to 
remain  in  sequence  as  they  pass  the  X.75  boundary. 

Fig.  10(b)  and  (c)  shows  variants  of  the  CCITT  intercon¬ 
nection  strategy.  In  Fqj.  l(Xb),  we  see  an  example  in  which 
only  X.25  is  used  both  as  a  network  access  method  and  as  a 
means  of  passing  traffic  across  network  boundaries.  A  single 
subscriber  or  a  pair  of  subscribers  to  two  nets  could  interface 
to  their  networks  via  X.25  and  to  each  other  by  means  of 
some  agreed  and  possibly  private  protocol. 

Virtual  circuits  would  be  explicitly  set  up  from  source  host 
to  gateway,  gateway  to  gateway,  and  gateway  to  destination 
host.  The  "internet"  addresses  of  the  source  and  destination 
hosts  could  be  carried  in  the  so-called  "Call  User  Data  Field" 
of  an  X.2S  Call  Request  packet.  This  leaves  the  packet  address 
field  free  to  identify  intermediate  destinations  (eq|.,  gateways). 


but  preserves  an  ultimate  internetwork  source/destination  ad¬ 
dress  which  the  gateway  can  use  to  select  the  destination  to 
which  the  next  intermediate  virtual  cucuit  is  to  be  set  up. 

An  alternative  to  this  is  shown  in  Fig.  KXc)  in  which  the 
subnets  A  and  ft  use  nonstandard  vutual  cucuit  interfaces,  but 
agree  to  build  gateway  software  employing  X  75  signaling  pro¬ 
cedures  across  the  gateway  interface.  This  solution  is  substan¬ 
tially  the  same  as  that  shown  in  Fig.  10(b),  except  there  is 
now  additional  translation  software  in  each  gateway  half  to 
make  each  virtual  circuit  net  work- access  protocol  compatible 
with  X.75  procedures. 

There  are  some  specific  problems  with  the  X.25/X.75  gate¬ 
way  strategy,  which  do  not  necessarily  apply  to  other  vutual 
call  gateways  The  basic  X.25  interface  provides  for  the 

sequence  numbering  of  subscriber  packets  mod.  8  or,  option¬ 
ally.  mod.  128.  Since  X.25  is  an  interface  specification,  this 
numbering  can  only  be  relied  upon  to  have  local  significance 
ti  e.,  hosl-to-packet  switch).  Some  X.25  implementations  use 
these  host-assigned  sequence  numbers  on  an  end-to-end  basis. 
Otheis  generate  internal,  net  work -supphed  numbers  to  allow 
for  repacking  ot  subscriber  packets  into  larger  or  smaller  units 
for  transport  to  the  destination.  If  packet  sequence  numbers 
assigned  by  the  source  host  were  carried  transparently  to  the 
destination  without  change,  it  might  be  possible  to  allow 
packets  to  (low  ou(-of-oider  across  the  X.75  boundary  to  a 
gateway  and  thence  into  the  next  network.  If  the  packet  se¬ 
quence  numbers  were  still  intact,  they  could  be  carried  out-of- 
order  to  the  next  destination  which  might  either  be  a  gateway 
or  an  X  25  host  In  the  latter  case,  the  original  packet-sequence 
numbers  could  be  used  to  resequence  the  packets  before  de¬ 
livery.  If  the  packets  were  being  delivered  to  an  intermediate 
gateway,  they  would  not  have  to  be  sequenced  there.  How¬ 
ever,  the  X.25  interface  specification  does  not  undertake  to 
carry  the  host-supplied  sequence  numbers  to  the  destination 
gateway  or  host  in  a  transparent  fashion,  primarily  so  that  the 
subnetwork  can  deal  more  freely  with  the  physical  packaging 
of  the  packet  stieani.  For  example,  a  source  may  supply 
packets  ol  length  1  28  bytes  while  a  destination  may  prefer  to 
receive  packets  no  longer  than  64  bytes.  To  allow  for  such 
variations,  the  network  must  be  free  to  renumber  packets  for 
delivery.  These  considerations  have  two  consequences. 

1)  X.25  packet  sequence  numbers  cannot  be  relied  on  for 
end-to-end  signaling,  though  they  could  be  so  used  if  requisite 
information  is  known  about  the  intermediate  transit  networks. 

2)  Packets  must  be  delivered  in  sequence  when  passing  to  or 
from  gateways  and  hosts  on  X.25  networks. 

The  second  conclusion  may  be  modified  slightly.  It  is  at 
least  es.sential  that  packets  be  delivered  in  relative  sequence  on 
each  virtual  circuit.  By  maintaining  independent  sequence 
numbering  on  each  virtual  circuit,  it  is  possible  for  hosts  and 
gateways  to  refuse  traffic  on  one  virtual  circuit  while  accepting 
traffic  on  another.  There  are  two  penalties  for  this.  First,  a 
gateway  must  keep  track  of  which  virtual  circuits  are  passing 
through  it.  Second,  dynamic  alternate  routing  of  packets  be¬ 
longing  to  the  same  virtual  circuit  through  alternate  gateways 
IS  not  possible  without  resetting  or  cleanng  the  virtual  circuit. 
This  last  point  is  simply  the  consequence  of  not  defining  an 
end-to-end  sequence  numbering  scheme,  but  instead  relying  on 
sequencing  of  the  packets  of  a  virtual  circuit  on  entry  to  and 
exit  from  each  intermediate  network. 

Some  networks  implement  X.25  level  acknowledgments 
(i.e.,  level  3)  that  have  an  end-to-end  significance,  but  others 
make  this  purely  a  host-to-paeket  switch  matter.  As  a  eonse 
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tiucnce,  It  IS  not  possible  to  rely  on  X.2S  packet  acknowlcdK- 
ments  to  determine  which,  if  any,  packets  were  not  delivered 
as  a  result  of  the  resetting  or  clearing  of  a  virtual  cu'cuit.  Fur¬ 
thermore,  even  if  a  subnet  were  to  offer  an  end-to-end  ac¬ 
knowledgment  between  a  source  host  and  an  X.7Sgateway,  this 
could  not  be  assumed  to  guarantee  that  the  acknowledged 
packet  was  delivered  to  the  ultimate  X.  25  destination  in  another 
network. 

X,7S  is  an  interface  intended  for  use  between  public  net¬ 
works.  Thus,  It  IS  not  likely  to  be  used  or  even  allowed  as  an 
interface  between  public  networks  and  private  networks.  For 
the  case  illustrated  in  Fig.  14,  X.2S  interfaces  could  be  pro¬ 
vided  between  public  and  private  networks  tor  other  special 
interfaces)  and  X.7S  interfaces  between  public  networks.  Con¬ 
sequently,  gateways  between  public  and  private  networks  are 
likely  to  appear  to  be  ordinary  host  computers  in  the  view  of 
the  public  networks. 

The  use  of  X.2S  for  private/public  network  interfaces  and 
X.75  for  public/public  network  interfaces  leads  to  the  situa¬ 
tion  shown  in  Fig.  14  in  which  an  internetwork  virtual  circuit 
would  have  to  be  made  up  of  several  concatenated  parts  such 
as  virtual  circuits  1-2- 3-4  (see  also  |S2,  Fig.  3.4) ).  Even  if 
X.2S  implementations  uniformly  permitted  an  end-to-end 
interpretation  of  packet  sequence  numbers  and  acknowledg¬ 
ments,  there  would  still  be  separate  virtual  circuits  required 
between  the  source  or  destination  hosts  and  the  gateways  into 
the  public  networks.  However,  the  concatenation  of  virtual 
circuits  docs  not  yield  a  virtual  circuit.  For  instance,  a  gate¬ 
way  between  the  public  and  private  net  could  acknowledge  a 
packet  but  fail  to  get  it  delivered,  in  which  case  the  subscriber 
will  have  been  misinformed  as  to  the  delivery  of  the  packet. 
This  situation  forces  the  end  subsenbets  of  private  networks  to 
implement  end-to-end  procedures  on  top  of  any  concatenated 
vutual  circuits  provided  by  the  public  networks. 

VIII.  Practical  Nktwork  Connections  ani> 
F.xpkrimfnts  in  Progress 

A  number  of  networks  have  been  connected  successfully 
over  the  last  few  years.  Most  of  these  connections  have  been 
made  in  an  aJ  hoc  manner,  using  one  of  the  following  tech¬ 
niques. 

1)  One  network  is  a  star  network  with  remote  RJF  and  in¬ 
teractive  stations  The  other  is  a  star  or  distributed  network 


with  clearly  defined  pnitocols.  A  device  on  the  star  network 
provides  exactly  the  functions  required  by  its  own  network  on 
one  side,  and  those  of  the  other  network  on  the  othei  side 

2)  Formal  gateway  s  are  provided  between  the  two  networks, 
and  protocol  mapping  occurs  in  the  gateway  . 

3)  A  computer  is  a  host  on  two  networks  It  is  arranged 
that  services  arc  provided  by  accepting  input  from  one  net¬ 
work  and  putting  it  out  on  another,  possibly  after  substantial 
processing. 

4)  Formal  gateways  are  provided  between  the  two  networks. 
Sufficient  agreement  is  obtained  that  end-to-end  protocols 
teven  high  level  ones)  are  common  in  Hie  two  networks.  In 
this  case,  less  activity  is  required  in  the  gateway. 

In  the  first  method,  a  form  of  front-end  computer  is  used. 
It  has  been  adopted  in  the  large  airline  and  banking  networks 
SITA  |13|  and  SWIFT  (14|.  In  each  case  the  standards  for 
the  networks  have  been  defined  rigidly.  SWIFT  has  even  certi¬ 
fied  officially  the  devices  of  three  manufacturers  to  provide 
interfaces  to  its  network.  The  other  side  of  the  device  is  then 
programmed  to  meet  the  requirements  of  the  star  system  being 
attached.  In  the  two  cases  cited,  only  a  simple  message  level 
of  interface  needed  to  be  defined. 

Other  examples  of  the  same  technique  are  the  connection  of 
the  Rutherford  laboratory  (Rl)  star  system  1531  and  the 
Livermore  CTRNET  to  ARPANET.  In  these  examples,  more 
serious  protocol  mapping  was  required  ARPANE'V  has  a  well- 
defined  set  of  HOST-IMP,  HOST-HOST.  Virtual  Terminal,  and 
File  Transfer  protocols.  All  these  had  to  be  mapped  into  the 
appropriate  procedures  for  the  other  network. 

The  second  method  has  been  applied  only  expeiimcnially. 
The  IK'L  interface  between  ARPANE'T  and  the  I'XPost  Office 
Experimental  Packet  Switched  Service  (EPSS,  |55|)and  the 
National  Physical  Laboratory  inteifacc  between  F.PSS  and  the 
European  Informatics  Network  (FIN,  (561)  are  examples  of 
this  technique;  a  demonstration  has  even  been  made  of  FIN- 
FPSS-ARPANET  with  no  extra  problems  cnciiuntcrcd  from 
the  three  networks  being  concatenated.  Technically  there  is 
almost  no  difference  between  the  first  two  methods.  The  sei"- 
ond  looks  at  first  sight  somewhat  more  general  than  the  first, 
hut  almost  the  same  problems  have  to  be  overcome.  The  diffi¬ 
culties  come  from  the  fundamental  differences  in  the  design 
choices  made  in  the  protocols  of  the  differeni  networks,  these 
differences  are  in  general  difficult,  and  even  sometimes  impos¬ 
sible,  to  resolve  completely.  In  the  fust  method,  they  can 
sometimes  be  resolved  using  a  specific  facility  in  the  star  net¬ 
work.  in  the  second,  where  two  distributed  networks  are  in¬ 
volved,  this  recoursi"  may  no  longer  be  available. 

One  example  of  the  problem  occurs  in  the  connection  of 
EPSS  and  ARPANFT.  ARPANFT  can  forward  any  number 
of  characters  at  a  time,  and  often  uses  full  duplex  remotcecho- 
ing.  FPSS  works  in  a  half-duplex  mode,  forwarding  only  conE 
plete  records.  A  special  "Transiiiil  Now"  has  to  he  input  by 
the  user,  and  interpreted  by  the  gateway,  to  ensure  that  partial 
records  are  forwarded.  Another  example,  from  the  same  appli¬ 
cation,  occurs  in  File  Transfer.  ARPANFT  assumes  an  inter¬ 
active  process  is  live  throughout  the  file  transfer,  all  comple¬ 
tion  codes  arc  passed  over  this  live  channel.  The  RL  network 
(and  FPS.S)  assume  that  file  transfer  is  a  batch  ptoerss,  they 
return  network  completion  codes  at  a  later  time,  and  may 
delay  acting  on  the  commands.  With  the  ARPANFT-RL  link 
(53) ,  the  file  transfer  job  had  to  he  given  a  very  hqtb  priority, 
so  that  (he  completion  covte  usually  arrived  before  a  timeout 
occurred;  because  of  the  nature  of  the  way  the  computer  was 
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u^i'J  tor  UtKi'  ri'jl-liini.'  johs,  tins  ilul  not  jlwjys  iii'.utc  (tut 
the  |oh  was  tun  in  a  riasonablc  tiim 

I'hero  arc  several  examples  ot  tlie  ihitJ  leslinit|Ue.  A  l)l'( 
PDI*  10  machine  used  on  the  Stantord  bniversity  SLIM!  X 
project  IS  a  host  both  on  AKl’ANl-  I  and  on  l  YMNl  T.  several 
machines  at  Boll.  Beranek  and  Newman  are  both  on  AKPANF T 
and  I  bLi-Ni  T  Because  the  Tl  NbX  operatiriK  syslein  has 
Kood  facilities  tor  linking  between  programs,  it  would  be  pos¬ 
sible  for  interactive  streams  to  come  in  one  network  and  go 
out  on  another.  File  transfer  problems  would  be  simple  in  this 
configuration,  because  the  hosts  obey  all  the  conventions,  in 
any  case,  of  each  network.  Of  course,  this  mode  of  operation 
may  require  that  files  in  transit  between  networks  may  have  to 
be  stored  temporarily  in  their  entirety  in  the  host  serving  as 
the  gateway  between  the  networks. 

1'he  fourth  technique  is  newer,  and  has  many  variations.  As 
a  result  of  agreement  on  the  X.2S.  and  partial  agreement  on 
the  X.75,  protocols,  PTT  networks  arc  able  to  interconnect  in 
a  reasonably  straightforward  manner.  The  connections  between 
DATAPAC  and  both  TFLENliT  and  TYMNET  have  been  done 
in  this  way.  In  each  case,  there  has  not  been  any  agreement  on 
higher  level  protocols,  so  the  problems  of  host-host  communi¬ 
cation  across  concatenated  networks  is  nut  resolved  by  these 
linkups  of  the  subnets. 

The  ARPA-sponsored  INTERNET  project  has  tried  to  stan- 
dardire  to  a  higher  level.  A  host-host  protocol  has  been  defined 
(TCP.  (  25 1 ),  and  is  being  implemented  on  a  number  of  differ¬ 
ent  networks  including  Packet  Radio  I20|,  (21 1,  ETHERNET 
(I8|.  LCSNET  1641  and  the  SATNET  (221,  in  addition  to 
ARPANET  This  protocol  is  defined  for  use  across  networks, 
thus  each  packet  includes  an  “Internet  Header"  which  is  kept 
invariant  as  the  packet  crosses  the  different  networks.  One 
aspect  of  the  INTERNET  program  is  to  develop  gateways 
which  can  interpret  this  header  appropriately. 

By  late  1976,  the  ARPA  project  had  connected  together  the 
Packet  Radio  "Network,  the  ARPANET,  and  the  Atlantic  Packet 
Satellite  Network  using  two  gateways  bet  ween  the  Packet  Radio 
Network  and  the  ARPANET  and  three  gateways  between  the 
ARPANET  and  Packet  Satellite  Network.  It  is  routinely  pos¬ 
sible  to  access  ARPANET  computing  resources  via  either  of 
the  other  nets  and  to  artificially  route  traffic  through  multiple 
nets  to  test  the  impact  on  performance.  In  one  such  test,  a 
user  in  a  mobile  van  in  the  San  Francisco  area  accessed  a  DEC 
PDP-10  TENEX  system  at  the  University  of  Southern  Califor¬ 
nia's  Information  Sciences  Institute  over  the  following  path: 

1)  from  van  to  the  first  gateway  into  ARPANET  via  the 
Packet  Radio  Network, 

2)  across  the  ARPANET  to  a  second  gateway  in  London, 
using  a  satellite  link  internal  to  the  ARPANET, 

3)  across  the  Atlantic  Satellite  Network  to  a  third  gateway 
in  Boston; 

4)  across  the  ARPANET  again  to  USC-ISI. 

The  user  and  server  were  400  geographical  miles  apart,  but  the 
communication  path  was  50000  miles  long  and  passed  through 
three  gateways  and  four  networks.  Except  for  a  slightly  in¬ 
creased  round-trip  delay  time,  service  was  equivalent  toaduect 
path  through  the  ARPANET.  Since  the  Packet  Radio  Network 
IS  potentially  lossy,  can  duplicate  packets,  andean  deliver  pack¬ 
ets  out  of  order,  the  end/end  TCP  protocol  was  used  to  exer¬ 
cise  flow  and  error  control  on  an  end-to-end  basis.  The  avail¬ 
ability  of  a  common  set  of  host-level  protocols  substantially 
aided  the  ease  with  which  this  test  could  be  conducted. 


I  he  AKl’.A  I  ii'iiel  also  has  higli  level  slandaid  proloeols  al 
ready  iii  exisleiue  Ui  sui>poit  lile  lianstei  anil  viilual  teniiinalv 
(the  I  IP  and  IITM  I  pnilm-ols  |40|).  and  Ihese  aie  being 
letiolilted  above  Hie  inleinel  It  P  piotocol  lo  provide  a  stall 
daid  high  level  iiileinel  work  pioiocol  hierarchy 


IX  RE«ii>iAroK\  ls.siirs 

I'he  regulaloiy  issues  in  the  interconnection  of  packet  net¬ 
works  lakes  a  different  form  in  North  America  than  elsewhere. 
It  IS  hard  in  a  paper  of  this  type  lo  more  than  touch  on  some 
of  the  problems  involved,  l  ire  di.scussion  here  is  simplistic  in 
the  extreme,  and  no  attempt  is  made  to  put  the  issues  in  the 
legalistic  language  they  really  require. 

In  almost  all  countries  the  provision  of  long  distance  com¬ 
munication  transmission  and  switching  is  provided  by  a  regu¬ 
lated  earner.  In  most  countries  outside  North  America,  this 
carrier  is  a  single  national  entity  called  the  “PTT".  In  some 
countries  te.g.,  Italy)  there  are  different  carriers  for  different 
services  - e.g.,  telegraph,  telephone,  intercity,  mternational 
telephone,  etc.  In  North  America  there  are  many  carriers. 
Usually  only  one  in  each  geographical  area  has  a  monopoly  on 
public  switched  voice  traffic.  Also  the  so-called  “Record  Car¬ 
riers”  have  some  sort  of  monopoly  on  “record  traffic,”  which 
is  message  traffic.  In  a  “Value  Added  Network”  (VAN),  the 
operators  rent  transmission  equipment  from  the  carriers,  and 
then  add  their  own  switching  equipment.  These  VAN’s  are 
themselves  regulated  in  what  they  may  do,  what  traffic  they 
may  carry,  and  what  rates  they  may  charge.  Between  North 
America  and  Europe,  specific  "International  Record  Carriers" 
(IRC)  have  monopoly  rights  on  data  and  message  transmission 

in  collaboration  with  the  appropriate  European  PTT’s.  The 
regulations  take  into  account  who  owns  the  hosts  and  termi¬ 
nals,  who  owns  (he  switches,  who  rents  the  transmission  lines, 
what  types  of  traffic  is  carried,  what  is  the  geographic  extent 
of  the  network,  and  what  is  the  technology  of  long  distance 
transmission. 

In  Fig.  15.  a  single  network  /V  is  sketched.  It  consists  of 
switches  S  and  transmission  lines  /  ;  these  together  are  called 
the  data  network,  /),\.  It  consists  also  of  terminals  T  and 
hosts  //;  the  exact  difference  between  a  terminal  and  a  host  is 
not  very  clear,  we  believe  it  is  assumed  that  terminals  mainly 
enter  and  retrieve  data  without  processing,  while  a  host  trans¬ 
forms  the  information  by  processing  This  definition  probably 
docs  not  meet  the  picture  of  modern  “intelligent  terminals." 
but  It  IS  always  hard  for  the  regulations  to  keep  up  with  the 
technology.  II  the  total  network  is  all  localized  in  one  site,  so 
that  no  communication  lines  cross  public  rights  of  way, 
then  It  can  usually  be  considered  from  a  regulatory  viewpoint, 
as  a  single  host  in  more  complex  network  connections.  The 
hosts  and  the  terminals  can  be  connected  to  the  switches,  and 
the  switches  to  each  other,  either  by  leased  lines,  or  by  the 
Public  Switched  Telephone  Network;  the  first  type  of  connec¬ 
tion  IS  called  a  Irasfil  connection,  the  second  sn'itchfd.  In  the 
subsequent  discussion  of  this  section,  the  term  "host"  will  in¬ 
clude  localized  networks.  In  general  we  will  assume  the  con¬ 
nections  between  the  switches  are  via  leased  lines,  it  that  is  not 
the  case,  the  regulations  are  much  eased  in  general  (though  in 
some  countries,  like  Brazil,  no  data  transmission  is  permitted 
al  all  via  switched  telephone  lines). 

If  all  the  hosts  and  switches  arc  owned  by  one  organization 
P,  which  also  leases  the  lines,  then  P  is  said  to  own  and  operate 
the  network,  and  it  is  called  a  “Private  Network."  There  are 
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nolMOiWi  .4  anj  H  arc  connected.  Network  I's  practice;!  may 
break  country  H'i  rcKulations.  and  yet  he  accessible  trom  coun¬ 
try  H  It  IS  this  class  of  problems  which  delayed  seriously  the 
permiSMOii  by  the  Swedish  Data  Inspectorate  Board  lor  Swedish 
banks  to  connect  then  networks  to  SWIH  I 

Secondly,  some  of  the  functions  of  networks  or  nateways 
legal  m  one  country  may  be  illegal  in  another  Thus  LI.S.  car¬ 
nets  are  not  permitted  to  do  data  processing  in  then  data  net¬ 
works.  no  such  considerations  apply  in  most  European  coun- 
tnes.  Some  of  the  protocol  translation  activities,  some  of  the 
message  proces.sing  activities,  and  some  of  the  high-level  ser¬ 
vices  (e  g  the  provision  of  niuUiaddiess  links)  may  well  be 
classed  as  "Data  Processing."  and  hence  be  illegal  in  the  U  S. 
In  interconnected  networks,  this  raises  the  possibility  that 
functions  can  be  carried  out  outside  the  jurisdiction  of  the 
country  in  which  the  operator  initiating  the  activity  is  sited, 
and  yet  which  is  illegal  in  that  country.  This  subject  is  treated 
rather  fully  elsewhere  [60| .  A  clear  example  of  this  is  the  use 
of  message  services  operated  by  TYMSHARK  and  CCA  on 
TtLtNFTand  TYMNtT.  While  these  services  are  legal  in  the 
U.S..  their  use  by  Uk  persons  connected  to  TYMNtT  by  the 
official  International  Packet  Switched  Service  is  clearly  tech¬ 
nically  illegal,  this  use  would  contravene  the  Uk  Post  Office 
Monopoly . 

X  I’NRFSiyLVVD  RKSFARIH  QVIKSTIONS 

There  are  many  unresolved  research  questions,  on  some  of 
them  even  the  present  authors  do  not  agree  with  each  other! 
Primarily  these  questions  have  a  technical,  policy,  administra¬ 
tive,  economic,  regulatory,  or  operational  aspect,  or  a  combi¬ 
nation  of  these. 

One  example  of  this  is  the  question  of  the  procedures  to  be 
used  for  internet  routing.  Here  there  are  technical  questions 
on  what  is  feasible  in  view  of  the  technologies  used  in  the  sub¬ 
nets.  there  are  policy  questions  on  when  third  country  touting 
might  be  allowed,  there  are  economic  considerations  on  how- 
much  It  would  cost  to  do  the  necessary  protocol  translation  to 
route  through  third  countries,  and  on  what  charges  the  con¬ 
necting  transit  network  might  make,  there  may  be  regulatory 
questions  on  which  clas.ses  of  data  may  flow  through  specific- 
countries  (related  to  the  transnational  data  flow  regulations); 
and  there  may  be  operational  questions  on  whether  in  the  event 
of  failure  in  dynamic  rerouting,  reestablishment  could  take 
place  with  sufficient  rapidity. 

Among  the  outstanding  research  questions  are,  in  alphabetic- 
order,  the  following 

Acccs.1  Control  What  are  the  requirements  and  methods  of 
implementation  of  access  control?  How  should  they  affect 
internetwork  routing? 

Adilrcssmg  How  should  the  International  Numbering  Plan, 
which  gcK’s  to  the  level  of  known  subscribers  of  public 
networks,  be  extended?  Should  this  extension  be  in  the  num¬ 
bering  plan  Itself,  or  should  additional  user  and  network  in 
formation  be  supplied?  Should  there  be  Uk-*1,  sir  only  physical, 
addressing?  Should  there  be  internetwork  source  routing 
implied  by  the  addressing? 

Broadcast  Farilitirs:  What  is  the  role  of  broadcast  communi¬ 
cation  facilities  in  the  provision  of  internet  services?  Should 
facilities  using  it  be  offered?  Should  technologies  supporting 
It  u.se  It,  particularly  at  gateways?  What  are  the  implications 
on  protocols,  especially  with  respect  to  duplicate  and  error 
detection’’ 

Datagram  versus  i'lrtual  Call  h'aetUues  How  should  data¬ 
gram  and  virtual  call  facilities  be  interconnected'’  How  can 


one  compare  the  relative  performance  and  costs  of  the  imple- 
mentaiions’’  Vkhal  criteria  should  be  used  in  any  aimparison’’ 
When  might  datagram,  or  alternatively  virtual  calls,  be  desir¬ 
able  or  essential  between  networks’’ 

Data  protection  What  are  the  effects  of  end-to-end  data 
encryption  on  protocol  Iranslatiun’’ 

Flow  and  Congestion  Control  1o  what  extent  should  one 
adopt  congestion  and  flow  control  between  gateways  and  their 
feedmg  networks,  between  gateways  directly,  or  between  gate¬ 
ways  and  the  source’’  What  are  the  relative  effects  of  just  dis¬ 
carding  packets  in  gateways,  and  relying  on  the  end-to-end 
prvMocol  to  detect  and  compensate  for  this?  How  is  charging 
for  discarded  packets  arranged’’ 

High  Level  Protocols  There  are  still  many  questions  on 
what  should  be  standardired.  and  how  rigid  the  standards 
should  be.  To  what  extent  should  the  individual  networks 
support  common  standards,  and  to  what  extent  should  proto¬ 
col  translation  be  feasible  technically  or  attractive  economi¬ 
cally'’  What  are  the  exists  of  maintaining  standards  or  the 
economic  advantages  of  standard  hardware  and  software? 
How  does  the  technology  of  mdividual  networks  and  the 
proportion  of  internetwork  traffic  affect  the  decisions? 

Internetwork  Diagnosis:  There  arc  many  technical  problems 
in  isolating  faults  in  concatenated  networks.  There  are  also 
organizational  and  economic  problems  on  who  should  be 
responsible  for  their  repair,  and  how  costs  for  service  failures 
should  be  allocated. 

Performance:  How  do  choices  of  design  parameters,  and 
network  services,  affect  the  costs  of  the  individual  networks? 
How  do  the  individual  network  performances  and  costs  scale 
to  large  networks?  How  do  the  choices  affect  the  feasibility, 
costs  and  performance  of  the  gateways?  How  do  the  varia¬ 
tions  in  technology  or  choice  of  parameters  affect  the  perfor¬ 
mance  in  interconnected  networks? 

Kouting  Policies:  To  what  extent  and  when  should  adaptive 
routing  be  used  between  networks?  How  can  one  recover 
from  the  partitioning  of  a  single  network,  when  there  are  still 
routes  existing  by  going  through  other  networks?  How  should 
administrative  considerations  affect  routing  policies  between 
networks  (privacy  regulations,  economic  cxinsiderations  of 
internet  payments.  de.sire  to  provide  for  high  availability,  etc.)? 
When  is  a  hierarchical  organization  more  efficient  that  a  direct 
route  search? 

Services  What  services  are  needed  on  an  internetwork  level? 
Clearly  interactive  and  bulk  transport  services  must  be  sup¬ 
ported.  What  else  IS  needed?  Should  the  internetwork  facilities 
be  able  to  support  voice,  telemetry,  and  teleconferencing? 
What  is  the  cost  of  supporting  these  latter  services,  and  what 
IS  their  effect  on  other  facilities? 

\.2S  and  X.7S  and  Related  Recommendations.  Is  X. 25 
suitable  for  transaction  processing?  Are  the  present  datagram 
proposals  adequate?  How  should  X. 25  be  extended  for  inter¬ 
net  addressing'’  How  should  X.25/X.75  be  modified  to  allow 
the  connection  of  private  to  public  networks,  or  private  net¬ 
works  to  each  other?  Do  the  X.3,  X.28,  X.2'4  pad  concepts 
extend  well  to  the  internet  environment,  or  should  they  be 
modified? 

XI.  CONCLUSiONS 

In  view  of  all  the  unresolved  questions  discussed  in  Section 
X.  most  of  the  conclusions  which  can  be  drawn  in  this  paper 
must  be  tentative.  From  the  early  part  of  the  paper,  we  have 
shown  that  it  is  essential  that  techniques  be  developed  for  con- 
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nccling  CDmputci  nctworki.  Moreo\fi,  no  Mnglc  sol  of  tcvh- 
niquc>  will  fit  all  j|'pli<.  Jtiona. 

The  Mtrvicca  wtiu'h  will  noiniall>  havr  to  br  auppoitcU  aic 
trrminal  bulk  tranafei.  remote  job  entry,  ami  tranaac 

tion  piov.'es>Mng  I'hc  vjuality  and  fa>;ilities  of  the  servieca  re- 
vjuired  will  be  very  JepenJenI  on  the  applieationa 

1  he  connectiona  between  networks  can  be  made  at  the  level 
of  Ihe  packet  switches  or  of  hosts,  and  can  be  on  a  datagram 
or  virtual  call  basis  Connection  at  the  packet-switch  level 
retjuues  broadly  similar  network  access  procedures,  or  com- 
ples  protocol  transformation  at  the  gateways  between  the 
networks.  If  the  network  protocols  are  different,  intercon¬ 
nection  can  be  most  easily  achieved  if  done  at  the  host  level. 
The  higher  levels  of  service  can  be  mapped  at  service  centers, 
which  need  not  be  colocated  with  the  gateways  but  very 
different  philosophies  of  network  services  can  be  very  difficult 
to  map  Alternatively,  subscribers  can  implement  common 
higher  level  protocols  if  these  can  be  agreed  upon. 

The  principal  problems  in  connecting  networks  ate  much  the 
same  as  those  in  the  design  of  the  individual  networks  of  het¬ 
erogeneous  systems  but  the  lack  of  a  single  controlling  au 
thorny  can  make  the  niullinet  design  problem  more  difficult 
to  solve  It  IS  essential  to  resolve  the  usual  problems  of  flow 
control,  congestion  control,  routing,  addre.ssing,  fault  recovriy, 
flesibility .  protiKol  standards,  and  economy  The  public  cat 
nets  have  attempted  to  resolve  many  of  these  problems,  par 
licularly  in  the  areas  of  flexibility,  addressing,  and  economy 
we  feel  theu  ssiliitions  are  not  yet  adequate  .At  the  higher 
levels  of  protocol,  much  more  standardization  is  required  be¬ 
fore  we  have  really  satisfactory  long  term  sc'lutions 

The  advent  of  international  coiiipuler  networks,  private  net 
works  which  must  communicate  with  other  private  networks 
(even  if  vu  public  ones),  and  the  new  applications  of  computer 
networks,  raise  regulatory  and  legal  issues  which  are  far  from 
resolution 

Many  technical  solutions  to  the  problems  of  the  connection 
of  networks  are  discussed  in  this  paper  Iheir  applicability  in 
view  of  the  different  technical,  economic,  and  policy  con 
straints  imposed  in  different  countries  must  still  be  assessed 
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CCITT  Recoreaendatlon  X.25  is  the  Internatloaal  standard  Interface  to  packat-mode  public  data 
networks  (PDN's).  CCITT  Draft  Reconmendatlon  X«75  defines  the  interface  between  packet-node  PDN'S. 
The  Interconnected  PDN's  will  comprise  an  international  public  data  network  system.  The  interface 
standards  and  the  system  architecture  that  they  imply  will  determine  Che  kinds  and  Qualities  of  ser¬ 
vice  that  will  be  available  to  subscribers.  This  paper  discusses  the  Interfaces  and  the  Implied 
system  architecture  In  the  light  of  subscribers'  requirements  and  of  the  characteristics  Inherent  In 
oacket  switching  technology.  K  brief  description  of  the  X.25  and  X.75  interfaces  Is  followed  bv  an 
evaluation  of  the  ''concatenated  segment"  architecture  that  they  Imply.  An  alternative  architecture, 
based  on  the  use  of  an  end-to-end  protocol  within  the  international  communication  system,  is  presen¬ 
ted  and  evaluated.  This  "DCE-DCE"  approach  seems  to  have  technical  advantages  over  the  concatenated 
segment  approach.  The  long-term  economic  Implications  need  to  be  examined. 


1 .  INTRODUCTION 

CCITT  Recommendation  X.23(I)  has  been  adopted  as  the 
international  standard  Interface  to  packet-mode  pub¬ 
lic  data  networks  (PDN's).  PDN's  offering  service 
through  this  interface  are  in  operation  or  in  con¬ 
struction  in  several  countries,  Including  the  United 
States,  Japan,  Canada,  United  Kingdom,  France,  Ger¬ 
many,  the  Nordic  countries,  and  Spain.  The  CCITT 
have  drafted  Recommendation  X.75(2)  to  define  the  in¬ 
terface  between  packet-mode  PDN'S.  When  the  inter¬ 
face  between  PDN's  has  been  adopted  and  implemented, 
the  interconnected  PDN's  will  comprise  an  interna¬ 
tional  public  data  network  system.  The  interface 
standards  and  the  system  architecture  that  they  imply 
will  determine  the  kinds  and  qualities  of  service 
that  will  be  available  to  subscribers.  This  paper 
discusses  the  interfaces  and  the  Implied  system  ar¬ 
chitecture  in  the  light  of  subscribers'  requirements 
and  of  tVie  characteristics  inherent  in  packet 
swl  tching  technology. 

2.  SUBSCRIBERS'  REQUIREMENTS 

Subscribers  require  a  unified  international  network 
system  that  meets  their  needs  for  both  local  and  in¬ 
ternational  data  communication.  The  system  must 
economically  handle  the  kinds  of  traffic  generated  by 
most  classes  of  subscribers.  It  must  provide  con¬ 
sistent  interfacing  procedures  that  are  independent 
of  the  locations  of  the  communicating  subscribers. 
And  it  must  provide  acceptable  throughput,  delay,  and 
error  characteristics. 

2 • I  Types  of  service 

Two  types  of  service  would  fulfill  the  requirements 
of  nearly  all  subscribers:  virtual  circuit  service 
and  transac tion-oriented  service. 


2.1.1  Virtual  circuit  service.  The  majority  of  sub¬ 
scribers  require  a  service  which  mimics  the  service 
currently  provided  by  circuit-switched  networks.  In 
this  service,  communication  is  established  and  broken 
relatively  infrequently,  and  communication  is  main¬ 
tained  for  relatively  long  periods  of  time.  Data 
must  be  delivered  in  order.  The  frequency  of  lost 
data  and  undetected  bit  errors  must  be  within  the 
range  that  the  subscriber  can  tolerate.  The  rate  at 
which  data  flows  must  be  controllable.  Service  with 
these  characteristics  is  called  "virtual  circuit" 
service . 

2«1«2  Transaction-oriented  service.  A  growing  number 
of  subscribers  require  a  service  which  supports  short 
transactions.  In  this  service,  the  communication 
path  need  not  be  maintained  between  transactions. 
The  transactions  are  independent  of  each  other,  and 
their  order  need  not  be  maintained  by  the  communica¬ 
tions  system.  The  subscribers'  equipment  is  designed 
to  handle  errors;  its  correct  operation  does  not 
depend  on  an  error-free  transmission  medium.  Data 
flow  between  any  two  points  is  relatively  infrequent 
and  therefore  need  not  be  controlled  by  the  transmis¬ 
sion  medium.  The  communication  system  can  discard 
data  when  the  flow  causes  congestion.  Service  with 
these  characteristics  is  called  "transaction- 
oriented"  service.  Potential  applications  of  this 
service  are  point-of-sale  terminals,  electronic  funds 
transfer,  distributed  data  base  management, 
telemetry,  and  speech  transmission. 

Virtual  circuit  service  and  transaction-oriented  ser¬ 
vice  can  be  implemented  in  a  number  of  ways.  The  way 
in  which  they  are  implemented  can  have  a  profound  ef¬ 
fect  on  many  aspects  of  the  communication  system,  as 
we  shall  see  later. 
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2  •  •’  Addfeitulfij} 

Thr  wav  in  which  one  subscriber  addresses  another 
through  the  public  data  network  system  should  be  the 
•4«r  whether  the  two  subscribers  are  located  on  the 
same  network  or  whether  they  are  located  on  different 
networks.  If  a  subscriber's  "terrainal"  Is  a  very  com¬ 
plex  one,  such  as  a  private  network,  the  addressing 
s  hete  should  permit  hlgh-resolutlon  "subaddressing". 

2  .  )  Rout  Ini^ 

Packet  switching  technology  provides  very  flexible 
means  tor  routing  calls  and  transactions  between  sub¬ 
scribers.  In  particular,  the  technology  allows  for 
alternate  routing  around  failed  or  congested  parts  of 
I  hr  t.  xsnunlcat ion  system,  and  even  for  dynamic  re¬ 
routing  without  Interruption  of  virtual  circuits. 
The  architecture  of  the  communication  system  should 
take  advantage  of  these  features. 

3.  INTERFACES  AND  ARCHITECTURE 

Fig.  1  shows  the  place  of  the  X.25  and  X.75  inter¬ 
faces  In  the  international  public  data  network  sys¬ 
tem,  X.25  defines  the  interface  between  the 
subscriber's  equipment  (Data  Terminal  Equipment,  or 
DTE)  and  the  PDN's  interfacing  equipment  (Data 
Circuit-terminating  Equipment,  or  DCE).  X.73  defines 
the  interface  between  the  Internetwork  interfacing 
equipment  (Signalling  Terminals,  or  STE's)  of  two 
PDN's. 


3 • I  Interface  structure 


Fig.  2  shows  the  three-level  structure  of  both  X.25 
and  X#75.  For  each  interface,  level  3  defines  how 
packets  control  and  effect  the  transmission  of  data 
between  subscribers.  For  each  interface,  levels  1 
and  2  define  the  physical  characteristics  and  pro¬ 
cedures,  respectively,  for  reliably  transmitting 
level  3  packets  across  the  interface.  CCITT  are  ur¬ 
gently  studying  the  procedures  for  using  multiple 
physical  circuits  between  STE's  to  increase  bandwidth 
and  reliability. 

3.2  Types  of  service 

Both  X.25  and  X.75  currently  provide  for  virtual- 
circuit  service  only.  CCITT  are  currently  studying 
the  addition  of  transaction-oriented  service. 

The  virtual  circuits  defined  by  X.25  and  X.75  provide 
flow-controlled,  order-maintained  communication 
between  the  DTE's  at  each  end  of  the  connection. 
They  also  provide  for  the  transmission  of  interrupts 
between  the  DTE's  and  for  notification  of  error  con¬ 
ditions,  which  may  involve  loss  of  data. 


DTE 

(STE) 


DCE 

ISTE) 


Fig.  2.  X.23/X.75  Inter¬ 

face  Structure 


3.3  Virtual  circuit  structure 


A  virtual  circuit  between  two  DTE's  that  are  connec¬ 
ted  to  different  PDN's  consists  of  several  concatena¬ 
ted  components: 

1.  the  interface,  defined  by  Recommendation 

X.25,  between  one  DTE  and  the  DCE  to  which  it 
Is  connected; 

2.  the  mechanism,  internal  to  the  first  PDN, 
that  implements  the  virtual  circuit  between 
that  DCE  and  the  STE  that  connects  the  first 
PDN  to  the  second  PDN; 

3.  the  interface,  defined  by  Recommendation 

X.75,  between  the  STE  in  the  first  PDN  and 
the  STE  in  the  second  PDN; 

4.  the  mechanism,  inrernal  to  the  second  PDN, 
which  implements  the  virtual  circuit  between 
the  second  STE  and  DCE;  and  finally 


PDN 


X.25 


Fig.  1.  International  Public  Data  Network  System 
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l?u'  itiCrrtiU'r,  d«*nnc»ii  hv  K*Ci>mm«»ndrti  ion 
X.*%,  ht’twi'H  l hr  N#i'fMui  iX’h  and  thr  DTK  IhAt 
I N  oonnrc  (rd  tt)  it* 

li  ttK-rr  intrrvtMi  iitg  PDN'h,  additional  i'omponcntN 

conni  Nt  ittx  ol  an  X./*)  Inlrriaor,  an  inlofSTK  mcohan- 
i  nm ,  and  am>lhrv  X./S  inlrriact"  arr  addrd  to  lh»'  l«- 
l»l«*«*M>tat  it»n  ol  tin*  vlrliul  circuit  lor  rach  intcr- 
vcninR  iM)N. 

Ii  is  im|s>i(an(  to  noir  that  the  liinctionM  to  hi'  ivr- 
loimi’d  hv  I  he  inttanelwoik  mechaniHnN  that  connect 
IH  > ' »  and  STT'-'n  are  not  deltned  by  tin*  CCITT. 

\ .  •'«  t)£^L  I  ona  I  I  ac  i  I  i  l  ie^ 

i*t>N  impl  eraiMit  ot'N  aie  not  reipiired  to  Implement  all  ol 
the  lactlitleH  detined  hv  the  IXITT  Htandardn.  Kxam- 
pi  ex  ol  thexe  optional  tacilllieK  are  reverae  char- 
^in>t  and  throoxhpnl  claati  t«o  lection. 

\ .  S  I’aramel^rw 

ri>N*S  are  permitted  to  vary  certain  iwirameterw  ol  the 
virtual  ctrcuiti)  they  implement  on  a  network-wide, 
KnhHcrlher-hv-Kuhncrther,  or  c i rcui t-hy-c 1 rco i t 
baxifi.  TlieHe  imrameterx  Include  the  maximum  ivicket 
xi/c  i>ermilted,  whether  reverse  charging  is  auppor- 
(ed,  and  no  torth. 

i.h  Addresning 

Thi'  xtandaidx  provide  lor  a  var  iahl  e-' leng  t  h  addrens 
tli.it  In  divided  into  a  network  tdentliier  and  an  In- 
tranetwork  addrcNN.  The  m.ixtmum  lengtii  ol  the  in- 

iranetwork  addrexN  (10  dlgiln)  In  loo  Nhort  in  Nome 

nelwoikn  to  provide  lor  hi gh- reNot ut t on  NuhaddreNNeH. 

\  ,  7  Hoot  I ng 

No  piocedore  tor  inteinelwork  routing  lx  deltned. 
Vhere  In  no  w.iy  lor  the  STK.  in  t  lie  originating  PON  to 
Nl'ecltv  1  Iw  route  lluil  a  cal  I  ix  (o  lake.  A  roeanx 
ti>i  deteimining  t  Iw*  loule  ol  a  compli'led  call  In  de- 
lined,  hut  tlw're  In  no  meanx  lor  determining  the 
route  taken  hv  an  unxiH'cexxt  ul  call  or  the  iHilut  at 
whith  t  Im  tail  tailed.  It  a  virtual  circuit  ix 

«  ie.ited  due  to  .i  failure  in  x«>me  PON  along  its  route, 
liiere  is  no  wav  tor  the  originating  STK  to  determine 
where  the  la  i  lure  occuired.  A  untipie  call  tdentiller 
In  provided  tor  each  call.  Hie  call  identitti'l'  lx 

intended  to  he  iixed  in  lallure  recovery,  hut  nt>  pro- 
ceduri*  lilt  thix  ix  vet  defined. 

4.  ADVANTAtiKS  t»K  THK  TCI TT  APPKOACM 

N  .  I  K  i  ow  t'ont  i  oi 

The  t'tlTT  .ipproai'h  provldex  fl«»w  control  at  evei  v 
point  along  e.ii  h  vlrtu.il  clrcull'x  (Mth.  TIitH  per- 
mltx  the  PDN'm  to  ettectively  conirol  hiiller  utilira- 
Iton  .ind  «* o  pieveni  c«»ngeNl  t .in • 


4.2  Communication  coNt 


The  CCITT  approacli  |H*rmltN  xhori  fiacket  Iwaderx  at 
the  ex|H*nNe  ot  proeexxing  and  m.ilntalnlng  Niatux  in- 
fvirmation.  Hut  conIn  ot  proci’XKiirN  and  roc*norv  are 
dropping  taNter  ih.in  liie  cok(  n  ol  common  teat  ion 
I  incN .  (  T) 

^  '  l^,- K  and  STV^  i mp I  eroent  a C  i  o^ 

X.^*!  and  X.7S  are  very  xiiuilar.  Thux  PDN  implcmen- 
tiirx  might  he  .-ihle  to  ciuixtiuci  STK'x  hv  ffliHii  lying 
DCK' N. 

4 . 4  Klexibi 1  tty 

The  fact  that  the  Nlandards  permit  optional  lacili- 
t  lex  and  implementation  (Mi'ameterN  glvcx  the  Im- 
pliNnentoi'N  a  great  deal  of  freedom. 

S.  DlSADVmAC.KS  OP  THK  CCITT  APPROACH 

Tile  C'CITT  approach  ti>  public  data  network  Nt.indardN 
is  limited  hv  the  concent  rat  Ion  on  interiaceN,  hy  the 
overall  ne(wi>rk  xlructure  that  ix  implied,  ,ind  hv 
looNenexH  ot  the  xtandardx. 

*>.1  (%>nc atenated  xej^mont  archlioctuve 

The  architecture  implied  hv  the  iutei faces  can  he 
c harac ! or i red  ax  a  concaienal ion  ol  IndeiHMUieni  xeg- 
rnentx.  Till*  virtual  circuit  service,  lot  inxlance,  lx 
implemented  hetwi*en  two  DTK's  hv ,  lirxl,  a  segment 
hetwi'en  the  the  DTK  and  the  IK‘K  to  which  it  is  con¬ 
nected,  followed  hv  Negmentx  hetvK'cu  the  >v\cket 
switches  that  make  up  the  PDN,  followed  hv  a  segment 
hetwi’iMi  the  iirxi  PDN's  STK  and  the  second  PDN's  STh' , 
etc.  Kacli  segment  ix  a  new  interlace.  I1u*re  is  no 
aspect  ol  the  service  wlUch  is  deltned  as  constant 
I  rom  oiu*  DTK  1 1»  l  he  other. 

Tills  approacli  reipiires  tiiat  I  he  entire  vlrtiuil  circu¬ 
it  service  must  he  implemented  in  eacli  segment. 
Kvi*ry  IXIK  aiul  SIT.  must  luindle  tlow  control,  i'itoi 
ciintrol,  and  all  tin*  other  as)x'C(s  ol  (lie  service. 
Thus  every  IX'K  or  STK  must  m.iintain  stale  inlormation 
about  every  virtual  circuit  that  passes  through  it. 
And  every  DO'  or  STK  nust  execute  the  relatively  com¬ 
plex  algorithms  required  (o  maintain  tlie  service. 

Tile  concatenated  segment  apptoacli  lias  t  lie  delect  that 
anv  data  loss  along  the  p.il  li  propagates  lliiiiugli  t  tie 
ciiain  with  uo  meaux  tor  irauspavent  vecovevv.  Tlie 
u'ldelecled  hit  error  rale  lor  a  vittu.il  circuit  is 
till*  sum  ol  I  tie  error  rales  in  eai  li  segmeni  tliat 
compr Ises  I t . 

*'•2  Vi  r  i  ua  I  lircull  utuieliiU'd 

In  the  tUMTT  appio.uli,  I  tie  luiictional  characteristics 
ol  I  lie  inti'aneiwork  segmeiils  ot  I  lie  archttectuie  ai  e 
tell  undefined.  Hut,  as  tig.  I  shows,  a  virtual  clr- 
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iitit  (onMi5(N  of  I'OlU  MirntUtul  Hi’KmiMU  K ,  Inclvuilng 
on»U*tinei1  int  r.^n«M  w»Mk  «  .  Thus  th«»  functton^l 

ohAtAv  tortsi  U  s  v>!  virtuAl  v  lrcull  hp  i^e- 

ttorJ  from  th«»  dettnttious  of  Its  »  Aud  thr 

loliiflonK  hetvpoo  thorn. 

tho  l/»ok  of  A  dottnoti  ii»ulin>;  j'rocpduro,  rttuf  of  a 
mortus  to  doloimtoo  fo  thr  svslom  a  or  a 

virtiutl  olioult  h%«  waWo  it  lm^^\^Rstblo  to 

roroulo  A  o.ill  or  virt\»rtl  circuit  ,irouud  a  (Allod 
olomoni  in  the  svstom,  ^^ain,  i  i\o  concrtton^iK^d  rcr- 
mont  ^  rc  1 1  m  i  uro  mAkos  it  difficult  to  r^t'ovor  tho 
stAto  ol  i ho  svstf'm  thAt  suppt'rts  u  virivjAl  circuit, 
h/’c.iuso  the  ini  otmt.i  t  i  «ut  th.At  ct>mprisos  t  ho  Rtrtto  of 
the  system  is  distrihuiotl  «>ver  a11  the  elemonis  on 
the  cUcuit's  ixtth.  U  a  rocovorv  procedure  using 
the  v»ni*;uo  c.tll  idonttflor  is  defined,  this  v>Tohlem 
mAV  he  solved. 

ReliAhlUlv 

The  veliAhiiitv  of  the  system  dpivnds  on  the  reliA- 
biiltv  of  e.tch  of  its  elements.  In  other  ^>rds,  l\^r 
system’s  relishilftv  Is  e‘iuAl  li.>  t  ix*'  relisbtlifv  oi 
IIS  MiMkest  element.  It  Is  for  this  resson  th.st  the 
implementors  of  the  P.tN's  hsve  g»>ne  to  giest  lengths 
to  construct  exiremelv  relisble  processors  snd  com¬ 
mon  (ca(  ion  lines  I  usiisllv  vio  redvindAnt  configui.i- 
l  i  ons . 

l.oos«'  stAndsrds 

PiiN  implementors  differ  in  their  interpretation  of 
the  stAndsrds,  snd  thus  in  the  detstls  of  u'fist  they 
hsve  implemented.  Pot  ex.tmple,  in  X.*^  the  flow  con¬ 
trol  mei-hsnism  is  defined  to  hAve  significance  only 
At  the  PTF-IX'K  interiAce.  Tiie  intTAnetv»rk  flow  con- 
lr»»l  hetweext  IX'P/s  is  not  defined.  Ixt  i.ACt,  flow 
coniii'i  is  defined  dllterentlv  in  different  PON's. 

s.xme  PON's  hAve  implemented  ’’snxchronous**  flow  con- 
tri'l:  e.tch  unit  of  flow  permission  given  hv  one  OTF 

to  Its  IX'F  is  mAtched  one-for-one  with  a  unit  of  flow 
permission  given  hv  the  txiher  IX'F  to  its  OTf .  Thus 
the  PON  Accepts  dstA  from  one  OTf  onlv  %dxen  the  other 
OTF.  hss  indiested  tts  reAdiness  to  receive,  'nils 
plA.es  the  burden  of  huffering  to  A«hleve  (hvovrghput 
f'li  t  lie  OTF  '  S  . 

Other  PON's  h.ive  tmplemeiUed  ‘’Asvnchri'nous"  1 1  ow  Ctin- 
t  rol  :  when  a  vtrtuAl  circuit  is  ostAbllshed  s*ilh  a 
given  throughput  clsss,  the  PON  reset  vos  entxxigii  in- 
tern.il  hiiffers  Along  the  circuit's  iNtih  To  ensure 
thrtl  the  given  throughput  Is  mAlntAined.  Tlie  PON 
gives  r.Xi'h  OTF  enough  flow  px'rmission  to  fill  the 
Pi'N's  buffers.  Thus  the  PON  relieves  the  OTK  of  sixme 
t)  I  the  burden  t>f  huffering  txx  Achieve  throughput. 

Wlxen  A  OTF.  built  for  Asvxtchronous  flow  control  is  At- 
tAched  t*>  A  PON  ihnt  provides  synchronous  flow  con- 
f rol ,  the  OTF  mnv  not  hAve  enough  huffering  to  mAln- 
f.iin  the  re.tulied  l  hroxighpxit  ,  Wlien  PON's  tliAf  employ 
iftlferent  ( U'w  control  schx'mes  ore  Interconnected ,  it 
is  n.»t  clear  what  the  result  will  he. 

Thus,  while  implemeniAiion  jvtrAmetevs  And  oplion.tl 
lac  titties  \v'rmH  ilexibUltv,  it  renuxins  to  be  seen 
how  well  dilterences  in  the  values  of  ihese 
pAtAmeters  can  he  niAde  trAitspArenf  to  subscribers. 


When  subscribers  c«xiinecied  to  dltfeient  PON  s  ihst 
use  different  psrsmeter  vslues  or  diftetent  inteipte' 
tAttons  Attempt  to  .  .sximxinii  ste ,  grAve  ptobiems  max 
I  esul  (  unless  the  the  subset  thei«  both  emplox  .1 
“lesst  cxxmmvxn  denominstor"  Interpi  ct  .it  ion  of  the 
KtAndsvds.  This  ApproA.'h  may  wx'rk  (>'i  the  subscit- 
hers.  Hut  it  wu'sns  lixst  they  must  choose  betwxpen  on 
the  one  hand,  loregoing  the  use  »xt  enhanced  facili- 
t  tes  that  mav  be  of  feted  bv  tlx*’ If  lo.al  PON  s ,  axxd  on 
the  other  hand,  employing  dttleient  pttx.eduies  on  lo 
caI  and  on  int  ernei  x.<xx  rk  vfttual  circuits. 

Ktnsllv,  ii  is  diiticult  to  see  how  a  PON  that  does 
Uv't  offet  A  given  optional  lacilitv  to  its  subscri 
bets  can  serve  as  an  intermedlarv  between  PON's  that 
dx>  ot  I  er  f  he  f  ac  1 1  1 1  v  . 

b.  AN  Al.TKRNATlVF  APPRdAt'H 

FAilier  packet  ss’itchtng  netvsMks  iAKPAMT,  t'Vi'l.AOFS, 
KlNl  employed  a  somewhat  diilriexxt  approach  lo  provi¬ 
ding  services  to  theit  subscvibexs.  Tins  Approach  Is 
called  the  "ond-io-envi**  Appioa.h, 

in  the  end-to-end  approach,  the  netwxxtk  provides  onlv 
A  very  basic  packet  switching  service,  Communicat ion 
between  subscribers  and  the  netwxxrk  is  on  a  ivtcket 
by-packet  basis.  It  the  subscribers  desiie  a 
t  ransAc  t  ion-OY  iex>\  ed  service,  thev  Mmplv  xise  the 
basic  txackel  swit.hing  seivice  as  it  is  provided  hv 
the  netwxxrk.  If  the  suhsi'ilbeis  desite  a  viiiusl 
circuit  service,  thev  implement  then  own  vtttvial 
circuits  between  them  via  an  "end-to-end  protocol". 
This  end-to-end  protx'x'x'l  uses  the  bast.  )>ACket 
switching  service  as  the  underIving  transmission 
medlvwi.  Virtual  circuit  establishment  snd  teimina- 
llvui,  evvxir  control,  and  flx'V  control  axe  all  haxidlexi 
hv  the  subscribers  at  the  end®  of  the  c.Mfimunical  ion 
path.  Hence  the  name  "xMxd-l  o-end" , 

Knd-lo-end  protocols  are  designed  with  axi  imj'etfecl 
.lata  t  ransmt  ssit'ii  medlisn  in  mind.  Kirors  such  as 
lost,  dupllcatx'd,  disordered,  or  altertM  data  ate 
handled  bv  txnhnfxiues  svxxh  as  vet  i  ansmi  ssi  on  and  re 
ordering,  Thev  thus  require  ot  the  transmission 
medfjOT  onlv  that  it  deliver  a  reasonable  fraction  of 
packets  tnfact . 

The  end-to-end  approach  could  be  applied  lo  the  in¬ 
ternational  pvibl  ic  dalA  neluMrV  structute.  .\s  showix 
in  tig.  4,  i  Iw  viviu.Al  civcxilts  coxild  he  implemented 
Via  an  end-to-end  piot.xcol  between  the  IVF's  at  each 
end  of  the  netwN'ik  cvrimunlcat  ion  ixath,  nils  could  be 
called  a  "IX'F-IX‘K"  approach,  Int  t  anri  s^x|  k  and  inter 
nell.^xrk  switching  wvuld  lake  place  on  a  ivicket-bv 
packet  basis.  Some  riiN’s  alreadv  w>xik  this  xesv  lx\- 
l evnal \ V , 

7.  AOVANTAvJFS  of  THF  IX'F-IX'F  ArrRxiAOH 
7,i  simple  impl emental ion 

The  1X»IX*V  approAx'h  w'uld  make  tl  unnecessary  toi 
evevv  IX'F  ov  STl-  to  implement  the  enitte  virtxial  cii- 
cuit  service  for  evevv  vivtu.1l  circuit  w'lxlch  jxasses 
l  hrotigli  it.  i>nlv  the  twv  IX'F's  at  ttw  ends  ot  a  vii 
lull  circuit  sMuld  have  to  mAintatn  state  iniox-mation 
about  the  virtxial  cticult.  And  onlv  IVF  %  wi'uld  have 
t.x  execute  the  |H‘r- vf  r  t  iia  I  clrcxiit  flow  .oiittol  and 
eiror  ontrol  alg.xr  1 1  Iwis ,  STF.'s  wxild  stmplv  pass 
packets  between  I'liS's, 
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DCE-DCE  END-END 


Fig.  4.  Alternative  System  Architecture 


^  .2  Reliability 

Failures  along  a  path  between  the  DCE's  at  the  ends 
of  a  virtual  circuit  would  only  seriously  affect  the 
virtual  circuit  If  all  paths  between  the  two  DCE's 
became  Inoperative.  Otherwise,  packets  which  may 
have  been  lost  through  the  failure  need  only  be  re¬ 
transmitted  by  an  alternate  route. 

7 . 3  Struc  ture 

The  DCE-DCE  approach  would  relieve  subscribers  of  the 
burden  of  Implementing  all  of  the  complexity  of  an 
end-to-end  protocol,  but  would  preserve  the  struc¬ 
tural  advantages  of  the  end-to-end  approach.  The 
characteristics  of  the  virtual  circuit  would  be  de¬ 
fined  all  along  its  path,  as  shown  in  fig.  5. 

Another  advantage  of  the  DCE-DCE  approach  Is  that  the 
International  transaction-oriented  service  could  be 
directly  supported  by  the  packet-switching  mechanism 
Internal  to  the  system. 

8.  DISADVANTAGES  OF  THE  DCE-DCE  APPROACH 

8.1  Complexity  of  protocol 

End-to-end  protocols  would  require  somewhat  more  com¬ 
plex  state  Information  and  algorithms, 

8 . 2  Flow  control 

Flow  control  would  not  be  provided  at  every  point  on 
a  per-vlrtual-clrcult  basis.  Other  mechanisms  would 
he  required  to  prevent  congestion  and  to  control  buf¬ 
fer  utilization. 

8.3  Communication  cost 

End-to-end  protocols  would  require  longer  packet 
headers  and  thus  higher  communications  costs. 


8.4  Flexibility 

The  DCE's  In  every  PDN  would  have  to  use  the  same 
end-to-end  protocol.  This  would  somewhat  reduce  Che 
flexibility  available  to  Implementors  of  PDN's. 

9.  CONCLUSION 

The  adoption  of  an  X.7S  standard  will  have  an  even 
greater  Impact  than  that  of  X.2S.  It  will  set  Che 
design  of  the  International  packet-mode  public  data 
network  system.  The  X.25  definition  left  open  the 
question  of  concatenated  segment  versus  end-to-end 
architecture.  The  current  X.75  definition  implies  a 
concatenated  segment  architecture.  This  architecture 
requires  extremely  high  reliability  components  that 
are  correspondingly  expensive.  Thus  the  interna¬ 
tional  public  data  network  system  may  fall  to  achieve 
the  economies  that  we  have  come  to  expect  from  packet 
switching. 

The  DCE-DCE  approach  Is  already  In  use  within  some 
PDN's.  If  it  were  extended  to  the  international  pub¬ 
lic  data  network  system,  the  desired  reliability 
could  be  achieved  at  a  significantly  lower  cost  for 
components.  Which  approach  would  give  the  best  ser¬ 
vice  to  subscribers  at  the  the  lowest  long-term  cost 
needs  to  be  determined. 
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ABSTRACT:  This  paper  examines  the  problems  of  using 
existing  transnetwork  architectures  to  support 
transnetuork  bulk  data  transfer.  TVo  architectures 
are  examined  In  some  detail:  the  DARPA  Catenet  and 
the  International  public  data  network.  Problems  of 
Integrating  flow  control  techniques  to  support  an 
end-to-end  service  are  encountered  with  both 
architectures.  An  experiment  Is  outlined  to 
investigate  these  problems. 


1.  Introduction 

Packet-switched,  data  communication  networks  were  originally 
developed  to  take  advantage  of  the  bursty  nature  of  much  of  the 
Interactive  traffic  exchanged  betwen  computers.  It  was  quickly 
recognised  that  statistical  multiplexing  of  data  packets  within  a 
communication  subnet  was  not  In  Itself  sufficient  to  handle  the 
requirements  of  all  users  In  a  satisfactory  fashion.  Different 
applications  have  different  requirements  from  the  medium  and  some 
applications  present  different  characteristics  to  It;  for  example. 
Interactive  transactions  typically  consist  of  packets  generated  at 
Irregular  Intervals  requiring  responses  within  a  fairly  short  time, 
«diile  some  telemetry  data  will  be  generated  very  regularly  and  may  be 
able  to  stand  a  certain  degree  of  data  being  lost  In  the  process  of  the 
session.  Hence,  different  techniques  are  required  for  handling  the 
traffic  in  order  to  meet  these  constraints  and  to  give  acceptable 
performance  to  the  application  process. 

This  paper  examines  one  particular  class  of  applications,  namely 
bulk  data  transfers  requiring  high  amounts  of  throughput.  The  aim  of 
the  paper  Is  to  assess  how  effectively  such  transfers  can  be  supported 
when  they  are  being  conducted  across  more  than  one  network.  Section  2 
of  the  paper  analyses  the  characteristics  of  bulk  dsta  traffic  relevant 
to  the  discussion,  and  briefly  surveys  some  of  the  techniques  used 
within  Individual  networks  to  handle  such  traffic  efficiently. 


2. 
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Another  paper  submitted  for  this  conference  [13]  examines  some 
particular  problems  in  transnet  flow  control  in  more  detail  using 
simulation.  However,  many  factors  Influence  the  rates  at  which  networks 
can  accept  and  deliver  traffic,  and  for  this  reason  an  architectural 
approach  to  flow  control  problems  which  Identifies  these  factors  and 
considers  their  Interactions  Is  also  needed.  This  paper  adopts  such  an 
approach.  Section  3  Is  a  general  study  of  the  areas  of  transnetwork 
connection  %rhlch  affect  transnet  flow,  and  considers  both  problem  areas 
and  the  architectural  framewrk  within  which  solutions  to  these  problems 
must  operate.  The  next  two  sections  are  case  studies  expanding  the 
arguments  developed  In  Section  3.  Section  k  assesses  the  DARPA  Catenet, 
an  approach  to  connecting  networks  which  has  been  evolved  by  a  research 
group  associated  with  the  Defense  Advanced  Research  Projects  Agency 
(DARPA).  Section  5  looks  at  the  Internetwork  architecture  evolving  In 
Public  Data  Networks  (PDNs)  as  a  result  of  the  adoption  of  CCITT 
Recommendations  X25  and  X7S. 

In  the  light  of  the  discussion  up  to  this  point,  a  transnet  file 
transfer  experiment  is  outlined  In  Section  6  which  will  Incorporate  both 
transnetwork  architectures  and  will  allow  us  to  examine  potential 
solutions  to  the  problems  encountered  In  a  real  transnetwork 
application.  Finally,  some  conclusions  are  dram  on  the  current  state 
of  transnetwork  flow  control  techniques. 


2.  Bulk  Data  Transfer  and  the  Single  Network 
2.1  Traffic  Characteristics  of  Bulk  Data  Transfers 

Of  the  major  network  user  services,  bulk  data  transfer  Is  the  one 
with  the  most  clearly  defined  set  of  requirements.  The  nature  of  the 
service  is  simply  defined:  It  is  to  move  a  given  body  of  data  from  point 
A  to  point  B.  The  amount  of  data.  Its  structure,  even  Its  content.  Is 
fixed  (in  most  cases)  at  the  time  that  the  request  for  the  transfer  was 
made.  Most  applications  requiring  bulk  data  transfer  are  Inherently 
very  Insensitive  to  timing  constraints,  unlike  Interactive  conversations 
or  many  transaction  applications,  but  do  require  that  the  structure  and 
content  of  the  data  be  preserved,  l.e.  that  nothing  Is  lost  and  that 
the  original  ordering  la  recovered  at  the  destination.  Because  of  this 
simplicity,  most  bulk  data  transfer  Implementations  itlll  adopt  a  very 
similar  approach  to  packet Islng  the  data  they  are  transferring  so  that 
efficient  end-to-end  flow  Is  achieved.  Therefore,  the  traffic  can  be 
characterised  by  the  following  tm  properties: 

1)  Nearly  all  the  data  packets  Involved  In  the  transfer  will 

be  large;  normally,  as  large  as  the  network  will  accept. 

2)  Packets  will  be  generated  frequently. 

The  rate  of  packet  generation  Is  often  subject  to  fairness  criteria 
within  the  host  operating  system,  and  Is  also  limited  by  the  amount  of 
traffic  the  network  flow  control  procedures  will  allow.  Nevertheless, 
one  can  state  the  following  proposition:  given  that  a  data  packet  from  a 
bulk  data  transfer  arrives  at  some  node  In  the  network,  there  is  a  high 
probability  that  another  such  packet  will  follow  very  shortly  after. 
This  probability  Is  affected  by  the  factors  mentioned,  and  Is  also 
higher  for  networks  with  fixed  routing  than  for  those  with  dynsmlc 
routing,  but  It  will  In  almost  all  cases  be  significant. 


BENNETT:  SUPPORTING  TRANSNET  BULK  DATA  TRANSFER 


Both  properties  are  needed  to  define  the  characteristics  a  bulk 
data  transfer  presents  to  a  network.  In  networks  where  the  user  pays  by 
nunber  of  packets  generated,  line  mode  terwlnal  conversations  will  be 
encouraged,  in  which  case  large  packets  nay  wll  be  generated  at 
relatively  Infrequent  Intervals.  On  the  other  hand,  many  telemetry 
applications  may  generate  small  packets  quite  often. 

It  has  been  noted  that  these  properties  are  more  suited  to 
circuit-switched  than  packet-switched  networks  (13].  Vihere 
packet-switched  netwrks  are  being  used,  it  is  important  that  they  be 
able  to  optimise  the  throughput  rates  for  bulk  data  transfer  connections 
where  short  transfer  times  are  desired.  To  achieve  this,  the  network 
should  fulfill  a  nmber  of  internal  subgoals: 

1)  Network  overhead  in  the  packet  should  be  minimised;  the 
packets  should  be  as  large  as  possible,  with  only  a  small 
portion  containing  control  Information  (packet  headers  and  the 
like) . 

2)  Packet  loss  rates  through  line  errors  and  node  congestion 
should  be  kept  as  small  as  possible,  both  on  end-to-end  and 
hop-by-hop  bases. 

3)  A  sufficient  nunber  of  buffers  should  be  allocated  both  at 
the  ends  and  at  intermediate  points  to  avoid  congestion  and 
deadlocks,  and  to  encourage  smooth  continuous  traffic  flow. 


2.2  Support  Techniques  within  Single  Nettrorks 

The  first  of  these  conditions  is  very  close  to  a  restatement  of  the 
first  characteristic  of  bulk  data  traffic  given  above;  within  the 
context  of  a  single  network  It  simply  means  that  choice  of  packet  sire 
is  the  responsibility  of  the  application  program.  We  shall  see  below 
(Section  3)  that  the  situation  is  not  quite  so  simple  when  ve  cone  to 
consider  multiple  networks  and  transnet  communication;  here  this 
condition  can  have  a  considerable  impact  on  flow. 

The  second  subgoal  Is  a  normal  design  requirement  for  communication 
networks  In  any  esse.  The  options  were  extensively  discussed  (at  least 
for  datagram  netwrks)  In  [25];  although  the  possibility  of  offering 
different  packet  loss  and  error  rates  as  a  user-selectable  option  has 
been  raised  (19],  these  options  have  neither  been  studied  fully  nor 
implemented . 

The  third  condition  implies  that  the  flow  control  techniques  used 
in  the  network  should  be  oriented  towards  the  needs  of  the  application 
traffic.  Ideally,  there  should  be  some  method  of  signalling  to  the 
network  that  a  given  traffic  stream  should  be  treated  as  bulk  data. 
This  is  nor  always  possible.  Many  networks,  such  as  the  ARPANET  [1] 
place  strict  limits  on  the  amount  of  buffer  space  that  can  be  made 
available  to  a  uaer  connection  within  the  network,  and  thla  ta  the  oome 
for  all  appl Icat Iona.  To  handle  bulk  data  tranafera  within  the  ARPANET, 
the  ALLOCatlon  technique  (24]  wma  Introduced;  in  thia  a  deatination  node 
would  automatically  reaerve  aome  buffer  space  for  future  traffic  from 
the  source  node  for  a  short  period  of  time  if  multi-packet  messages  were 
being  transferred.  However,  this  la  not  a  general  method  for  adapting 
automatically  to  the  nature  of  data  transfer  traffic,  as  It  relies  on 
im  aiKsiUT  machmta  for  Aarmrmctlva  cenvmrsations  arm 
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More  recent  datagraa  networks  are  building  explicit  awareness  of 
user  flow  control  requiresients  into  the  internal  network  architecture. 
The  trans-Atlantic  broadcast  satellite  network  SATNET  [20).  for  exasple. 
has  a  channel  protocol  tdiich  distinguishes  between  "block"  and  "atreaa" 
traffic,  and  will  reserve  slots  In  the  trananisslon  fraaes  on  either  a 
burst  basis  or  for  a  specified  tlae  period  accordingly.  Both  the 
traffic  type  and  various  associated  paraaeters  can  be  aade  known  to  the 
satellite  aessage  nodes  by  the  application  procesa  via  a  apeclal 
protocol . 

This  trend  is  also  observable  in  virtual  circuit  networka.  X25  [4] 
provides  a  definition  of  user-selectable  throughput  classes,  which 
represent  an  acceptance  rate  across  an  X2S  Interface.  This  aay  or  aay 
not  be  a  guarantee,  and  does  not  necessarily  carry  any  end-to-end 
iaplicat ions,  as  the  destination  Interface  aay  support  a  coapletely 
different  class  for  the  connection,  but  it  does  coauilt  the  network  to 
tying  up  buffers  specifically  for  this  connection.  In  the  case  of 
Datapac  [29],  the  terainatlng  network  node  (or  DCE)  will  in  fact  coaait 
aore  buffers  to  supporting  the  connection  than  the  end  host  (DTE). 


3.  Transnetwork  Flow  Control  Constraints 
3.1  Differences  Between  Networks 

When  Me  require  a  service  to  be  extended  across  several  networks, 
we  Introduce  new  variables  which  will  affect  our  ability  to  support  the 
service  efficiently.  These  all  arise  from  the  potential  differences 
between  networks,  and  aany  can  affect  the  flow  of  traffic  froa  source  to 
destination.  A  coaprehenslve  survey  of  the  current  probleas  In  transnet 
connection  is  given  in  [8];  only  those  factors  relevant  to  flow  control 
will  be  studied  here. 

i)  Differences  in  packet  size:  Networks  can  vary  greatly  in 
their  Internal  packet  sizes.  X2S  supports  a  range  of  aaxiaua 
packet  sizes  between  16  and  256  octets,  though  128  is 
recoaaended.  Full  ARPANET  aessages  can  be  1000  octets  long, 
while  soae  exper iaental ,  high  data-rate  networks  such  as  the 
Caabrldge  University  ring  net  [18]  have  packet  sizes  as  low  as 
4  octets.  These  different  sizes  aean  that  larger  packers  have 
to  be  fragaented  into  asaller  units  before  they  enter  the  net 
with  the  saaller  size;  for  efficiency  they  aay  have  to  be 
reasseabled  again  before  being  delivered  to  the  destination 
process  if  they  can  arrive  out  of  order  or  contain  an 
end-to-end  checksua.  The  creation  of  these  fragaents,  of 
course,  aeans  that  each  fragaent  will  carry  its  own  addressing 
inforaatlon  and  will  generate  its  own  network  overheads  - 
acknowledgeaents ,  retransmissions  etc.  In  particular,  buffers 
will  be  required  for  the  creation  of  the  fragaents  and  for 
reassembling  them  into  their  original  units.  Thus  the  way  in 
which  fragmentation  and  reassembly  is  lapleaented  can  directly 
affect  the  rates  at  which  the  end  processes  produce  and 
consuae  buffers. 

II)  Differences  In  service  characteristics:  The  differences 
between  adjacent  networks  caused  by  factors  such  as  the 
networks'  communication  media,  geographical  size,  or  aobllity, 
and  visible  in  such  characteristics  as  throughput  rates. 
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end-ro-end  delay,  and  error  and  loss  rates  can  all  affect 
transnet  flow  and  congestion  control  scheaes. 

ill)  Differences  in  user  options:  The  networks  may  be  virtual 
circuit  or  datagram  networks,  and  may  offer  different  degrees 
of  reliability.  Datagram  networks  may  not  preserve  ordering, 
in  which  case  the  gateway  or  the  end  process  may  have  to 
reserve  more  buffers  than  would  otherwise  be  necessary. 
Adjacent  networks  may  or  may  not  offer  user-controllable 
options  to  select  various  grades  of  service,  %«hich  may  differ 
between  adjacent  networks.  More  subtly,  they  may  conform  to  a 
common  specification  but  differ  significantly  In  the  details 
of  the  implementation,  as  in  some  X25  networks. 

3.2  The  Role  of  Gateways 

The  entity  which  must  perform  this  arbitration  between  the  adjacent 
net%#ork8  is  the  gateway  machlne(s).  A  standard  model  of  the  gateway 
[22]  sees  it  as  a  host  on  each  of  the  net%a>rks  it  is  connected  to.  In 
which  case  it  has  available  the  full  range  of  service  options  and 
network  limitations  of  an  ordinary  host  on  those  networks.  Provided 
that  it  Is  capable  of  distinguishing  the  needs  of  transnet  services,  it 
should  be  able  to  operate  these  to  support  service  across  the  local  nets 
as  though  it  were  a  local  host  to  host  service. 

The  factor  that  determines  the  degree  to  which  the  gatetiay  can  make 
Intelligent  local  flow  control  decisions  is  the  global  transnet 
framework  in  which  It  is  operating  in.  Several  possible  global 
architectures  are  Illustrated  the  protocol  layer  diagrams  of  Figure  1. 
If  there  Is  no  global  transnet  protocol,  then  the  gatewy  Is  required  to 
perform  a  complete  protocol  mapping;  In  this  case  the  gateway  can 
acquire  all  the  information  it  may  need,  but  the  service  It  can  give  Is 
severely  hampered  by  the  overhead  of  doing  the  mappings,  and  by  the 
degree  to  which  the  mappings  are  possible.  The  second  possibility  Is 
that  a  global  application  protocol  is  being  supported,  but  that  protocol 
mapping  is  required  betwen  transport  protocols.  Here,  the  problems  of 
protocol  mapping  should  be  simpler,  and  the  gateway  has  directly 
available  to  it  all  the  Information  about  options  that  the  application 
has  provided  to  the  transport  protocol. 

The  other  major  possibility  Is  that  the  system  is  fully  Integrated 
at  transnet  levels  -  i.e.  there  Is  a  global  transnet  transport 
protocol,  supporting  an  end-to-end  application  protocol.  In  this  case, 
the  capabilities  of  the  gateways  for  supporting  types  of  service  are 
largely  determined  by  the  nature  of  the  transport  protocol  and  the 
assumptions  it  makes  about  the  underlying  service.  As  with  local 
network  transport  protocols,  it  may  assisae  virtual  circuit  or  datagram 
service,  it  may  support  specifically  certain  types  of  service,  and  It 
may  be  able  to  communicate  such  information  to  the  gateways. 

Clearly,  the  gateways  will  play  a  major  part  in  supporting  the  flow 
pattern  experienced  by  a  particular  application.  It  also  becomes  clear 
that  the  subgoals  we  defined  In  Section  2  are  no  longer  entirely 
adequate  for  defining  transnet  support  procedures  at  gateway  level. 
Bearing  in  mind  the  differences  between  networks  discussed  sbove,  we  can 
redefine  these  subgoals  for  gatewsys  as  follows: 


1)  Support  maximum  size  fragments  with  minimum  overhead:  the 
gateways  should  minimise  fragmentation,  and  in  certain  cases 
may  perfom  Intamadlata  raaaaambiy. 
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a)  Protocol  Happing  Catatmy 


b)  Tranaporr  Mapping  Gateway 


c)  Network  Acceea  Gateway 
Figure  li  Fwidfntel  Trananet  Archlracturaa 
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2)  Minimise  packet  loss:  The  gateways  should  minimise  packet 
loss  at  the  gateways  and  choose  reliable  delivery  options  In 
local  networks.  The  local  network  should  be  able  to  handle 
loss  due  to  packet  errors;  losses  due  to  congestion  may  occur 
in  gateways  unless  proper  congestion  control  techniques  are 
Inst  1 1  uted . 

3)  Optimise  buffer  availability:  The  gateways  should  relate 
the  nunber  of  buffers  assigned  to  this  application  to  the 
expected  demand.  It  should  also  be  able  to  select  user 
options  from  the  local  network  on  the  basis  of  the 
applications  requirements.  The  end-to-end  transport 
mechanism,  where  applicable,  should  provide  the  gateways  with 
some  means  of  obtaining  this  Information. 


Tltese  aims  arc  similar  to  those  developed  In  Section  2,  but  apply 
to  the  gateway-to-gateway  level  of  transnet  architecture.  The 
'archetypal  properties'  of  bulk  data  transfer  protocols  also  defined  in 
Section  2  are  performance  goals  between  the  two  end  processes.  Thus  at 
each  layer  of  transnet  architecture  we  have  a  set  of  flow  control 
criteria  which  should  be  satisfied.  Each  set  can  interact  with  the 
others,  both  directly  and  indirectly,  and  each  requires  Information  from 
the  other  levels  to  provide  a  fully  efficient  service.  In  assessing  the 
effectiveness  of  flow  control  in  a  transnet  service,  therefore,  we  must 
examine  the  flow  control  techniques  at  each  level  to  see  how  well  they 
meet  the  criteria  of  maximum  data  efficiency,  minimum  packet  loss,  and 
optimal  buffer  usage.  We  also  need  to  examine  how  the  strategies  at 
different  layers  can  get  the  Information  they  need,  how  they  can 
communicate  Information  to  other  layers,  and  how  the  techniques  used  in 
different  layers  interact.  In  the  following  two  sections,  two 
transnetwork  architectures  are  assessed  In  this  light. 


4.  File  Transfer  in  the  DARPA  Catenet 
4.1  The  DARPA  Catenet  Architecture 

The  Transmission  Control  Protocol,  or  TCP,  is  a  network- independent 
transport  protocol  which  has  been  proposed  as  a  vehicle  for  transnet 
communication  since  it  was  first  formulated  in  1974  (6,  27].  Work  on  a 
transnet  architecture  (called  the  'Catenet'  [9])  has  been  continuing  in 
a  DARPA-sponsored  project  since  then,  with  major  evolutions  in  both  the 
TCP  and  the  theoretical  structure  of  the  Catenet.  It  should  be  noted 
that  the  association  between  the  TCP  and  the  Catenet  is  not  a  necessary 
one,  bur  the  two  projects  are  close  enough  to  be  regarded  as  one  for  the 
purposes  of  this  paper.  The  main  testbed  for  implementing  these  ideas 
has  been  DARPA' s  three  major  research  networks  -  ARPANET,  SATNET  [201 
and  PRNET  [21].  The  following  summary  represents  the  state  of 
development  of  the  model  at  the  time  of  %nrlting,  and  some  details  may  be 
out  of  date  by  the  time  of  the  conference,  but  in  essence  it  should  be 
unchanged . 

Figure  2  shows  the  basic  model  behind  the  DARPA  Catenet  [33]. 
Currently  the  Catenet  consists  principally  of  the  three  DARPA-rclated 
networks  given  above,  and  gateways  have  successfully  been  built  between 
ARPANET  and  SATNET  [20]  and  ARPANET  and  the  PRNET  [21],  Other  networks 
will  be  added  in  the  future.  The  networks  within  the  Catenet  are 
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a)  The  Future  UCL  Configuration  (slnpllfled) 
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connected  by  gateways  which  transfer  packets  from  one  net  to  the  next  on 
a  datagram  basis.  An  internet  packet  may  take  any  route  through  the 
Catenet,  it  may  be  fragmented  by  the  gate%rays  at  any  time,  and  each  of 
the  fragments  themselves  may  take  any  route,  to  be  reassembled  at  the 
destination  host  before  delivery  to  the  destination  transport  process. 
The  addressing  and  fragmentation  procedures  supporting  this  minimal  set 
of  facilities  is  defined  by  the  Internet  Datagram  Protocol  [28]  -  or 
IDP.  The  routing  procedure  is  currently  based  on  fixed  routing  tables, 
but  it  is  shortly  to  be  replaced  by  one  using  an  adaptive  algorithm 
[31].  The  IDP  makes  no  guarantee  of  reliable  or  sequenced  delivery,  and 
hence  the  gateway  congestion  control  currently  consists  of  a  simple 
blocking  technique  [7].  It  does,  howver,  provide  a  number  of  service 
options,  which  are  still  undergoing  refinement.  The  oldest  one  Is  a 
"Don't  Fragment"  option;  others  adopted  recently  Include  the  selection 
of  qualitative  delay  classes  (of  the  type  "As  fast  as  possible",  "Fast", 
"Normal",  "Don't  Care"),  and  the  creation  of  reliability  options  to 
facilitate  transmission  over  very  lossy  networks. 

Above  the  IDP  at  the  source  and  destination  hosts  comes  various 
Internet  transport  protocols.  The  most  ccxnpletely  defined,  and  the  most 
well  known  of  these  is  TCP,  which  will  be  at  the  basis  of  most 
conventional  network  services  such  as  virtual  terminals  and  file 
transfer.  Other  more  specialised  transport  protocols  such  as  a  Real 
Time  Protocol  (RTP)  are  envisaged  for  particular  applications  such  as 
packet  voice,  but  these  will  not  be  considered  further  In  this  paper. 

4.2  End-to-End  Flow  Control 

TCP  provides  reliable  sequenced  delivery  of  almost  arbitrarily 
large  Internet  packets.  The  end-to-end  flow  control  used  by  TCP  is 
based  on  a  windowing  scheme  using  octet-based  sequence  numbers  but 
incorporating  information  on  the  buffer  sizes  exchanged  at  set-up  time 
to  ensure  smooth  changes  In  window  size  [3]. 

Clearly,  when  small  window  are  in  use,  the  scheme  approximates  to 
the  sequenced  packet-by-packet  acknowledgement  schemes  required  for 
handling  interactive  traffic,  and  which  have  been  in  existence  for  many 
years.  When  large  windows  are  used,  the  situation  changes,  and 
acknowledgements  for  large  amounts  of  data  need  only  be  generated  at 
Infrequent  intervals  -  which  makes  windowing  schemes  Insensitive  to 
out-of-order  deliveries.  Windowing,  then,  comes  Into  Its  own  as  a 
technique  in  precisely  the  kind  of  situation  we  encounter  in  bulk  data 
transfer,  and  so  the  studies  which  have  been  made  of  various  TCP  flow 
control  policies  have  a  particular  significance  for  bulk  data  transfer. 

The  window  size  sent  by  the  receiver  to  the  sender  represents  a 
promise  of  buffer  space  availability  at  the  receiver's  end.  TVo 
policies  have  been  Identified  on  which  this  promise  can  be  based 
[32,14].  The  'conservative'  policy  uses  the  window  size  as  a  firm 
guarantee  of  buffers  actually  available  to  the  receiving  TCP  at  the  time 
It  generated  the  message.  The  'optimistic'  policy  bases  window  size 
calculations  on  some  predictive  heuristic  to  give  a  reasonable  estimate 
of  what  the  receiving  TCP  expects  to  possess  in  the  near  future.  As  one 
would  expect,  simulation  studies  [11]  have  shown  that  the  conservative 
policy  is  the  best  one  to  follow,  especially  when  buffer  space 
management  Is  the  responsibility  of  the  user  process,  which  Is  the 
policy  encouraged  by  the  TCP  implementors.  In  this  case,  an  optimistic 
policy  must  try  to  gauge  the  demands  that  the  user  process  Is  going  to 
make  on  the  TCP  and  the  resources  that  It  will  make  available.  The 
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current  Buggcstions  for  TCP  user  intcrfacea  do  not  provide  a  aeana  for 
doing  rhia  under  user  control,  although  such  an  Interface  could  be 
designed  without  violating  the  protocol.  End-to-end  congestion  control 
Is  based  on  retransmission  and  positive  acknowledgements. 
Retransmission  policies  have  been  investigated  in  [12];  this  study 
favours  a  policy  of  Increasing  the  retransmission  timeout  between 
retransmissions,  to  avoid  loading  the  Catenet  with  retransmissions. 

4.3  Gateway-to-gateway  Flow  Control 

Currently,  gateway-to-gateway  flow  control  Is  non-existent:  if  a 
gateway  cannot  accept  a  packet.  It  is  simply  dropped.  Since  the  packet 
is  an  internet  datagram,  its  eventual  arrival  at  the  destination  will 
depend  on  the  end-to-end  retransmission  of  the  packet.  This  policy  is 
acceptable  for  Interactive  traffic,  as  one  can  assume  that  the  gateways 
will  have  sufficient  time  to  clear  the  source  of  congestion,  or  cause  a 
routing  change,  before  the  next  packet  arrives.  Howver,  It  is  not  an 
acceptable  proposition  for  bulk  data  traffic,  since,  from  the  property 
we  established  in  Section  2,  the  discarding  of  one  packet  due  to 
congestion  will  probably  lead  to  the  discarding  of  several  subsequent 
packets  before  the  situation  is  corrected,  possibly  involving  a  large 
amount  of  data.  Congestion  is  more  likely  to  be  caused  by  bulk  data 
transfers  than  by  other  applications,  but  the  advance  buffer 
reservations  that  could  reduce  it  by  anticipation  cannot  be  made  as 
there  is  no  way  of  indicating  what  may  be  required. 

In  practice,  the  situation  may  be  even  more  complicated  than  this. 
While  gateway-to-gateway  delivery  may  not  be  reliable,  the  local  netwrk 
may  guarantee  reliable  delivery.  Because  there  is  no  gateway-to-gate%fay 
acknowledgement,  a  gateway  may  be  unable  to  distinguish  congestion 
within  the  network  from  congestion  in  the  neighbouring  gateway,  and  it 
certainly  cannot  tell  which  packets  are  being  delayed  because  of  It. 
Thus  the  situation  can  arise  whereby  a  gateway  is  injecting  an 
end-to-end  retransmission  into  a  network  which  is  still  trying  to 
deliver  the  earlier  transmissions.  This  situation  has  in  fact  been 
detected  in  TCP  experiments  across  the  ARPANET  [2]. 

It  is  not  clear  that  the  introduction  of  a  reliability  option  is  a 
solution  to  this  problem,  as  the  essential  features  of  the  solution  are 
that  the  gateway  must  be  able  to  detect  that  it  has  handled  an  earlier 
transmission  of  the  packet,  and  that  It  knows  why  the  earlier 
transmission  did  not  reach  the  neighbouring  gateway.  Both  features 
require  fixed  routing,  and  the  first  requires  that  there  is  a  defined 
coupling  between  the  end-to-end  sequence  numbers  and  the  fragment 
numbers  defined  in  the  IDP  for  use  in  the  gateways;  but  routing  is 
dynamic,  and  no  such  coupling  has  been  defined. 

Very  recently.  It  has  been  proposed  that  a  gate%Riy  detecting 
congestion  should  generate  an  advisory  message  requesting  the  quenching 
of  traffic  at  the  source.  Such  a  policy  could  alleviate  many  of  the 
problems  described  here,  but  It  raises  additional  problems  of  fairness, 
and  of  the  action  to  be  taken  if  such  messages  are  ignored. 

4.4  Effects  of  Fragmentation 

Another  aspect  of  gateway-to-gateway  functions  which  concerns  us 
here  Is  fragmentation.  Fragmentation  Is  a  procedure  which  Is  more 
likely  to  be  applied  to  bulk  data  transfer  packets  than  to  those  from 
other  applications  because  of  the  large  packet  sices  involved.  However, 
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the  overheads  of  fragmentation  are  large.  Each  TCP  packer  has  at  least 
16  octets  of  header  and  each  internet  fragment  will  have  at  least  a 
further  16,  in  addition  to  any  local  network  headers  that  must  be 
generated  for  each  fragment.  A  maximum  efficiency  of  77Z  is  possible  on 
the  ARPANET  for  internet  fragments  contained  within  a  single  ARPANET 
packet  128),  and  this  figure  drops  to  63Z  for  the  first  fragment  of  the 
packet  ,  carrying  the  TCP  header. 

The  disparity  in  maximum  packet  sizes  affects  the  processing 
efficiency  of  destination  hosts  in  more  subtle  ways.  A  reasonable 
buffering  strategy  at  a  receiving  host  in  a  bulk  data  transfer  would  be 
to  allocate  buffers  of  a  size  fixed  by  the  maximum  packet  size  of  the 
local  network.  Similarly,  a  reasonable  fragmentation  policy  is  to 
always  create  fragments  of  a  maximum  size  where  possible.  However,  the 
actual  pattern  of  fragment  sizes  created  by  this  policy  may  render  the 
buffer  consumption  very  inefficient.  For  example,  a  256  octet  internet 
packet  going  from  a  256  octet  network  Into  a  128  octet  net%x>rk  under 
this  scheme  will  be  fragmented  into  three  packers  -  the  last  containing 
16  octets  of  header  and  16  octets  of  data.  If  this  were  happening  in  a 
bulk  data  transfer,  then  the  receiver  would  be  processing  Incoming 
packets  three  times  as  fast  as  the  source  is  producing  them,  and  a  third 
of  the  packet  buffers  would  be  largely  empty.  This  renders  the  receiver 
very  sensitive  to  high  processing  loads  in  its  host,  directly  affecting 
end-to-end  throughput. 

Such  effects  can  occur  even  without  fragmentation.  In  the  example 
above,  a  file  being  transferred  in  the  reverse  direction  will  have  256 
octet  buffers  allocated  to  it  by  the  receiver,  and  every  one  of  these 
will  only  be  half  filled. 

Again,  because  the  routing  is  dynamic  and  the  communication 
datagram  based,  it  is  difficult  to  see  any  algorithm  guaranteeing  to 
avoid  these  effects  In  the  DARPA  Catenet.  Intermediate  reassembly  is 
not  an  option  supported  within  the  IDP,  although  it  is  recognised  that 
in  some  cases  a  private  gateway-to-gateway  scheme  may  be  necessary  [30]. 
This  Is  a  direct  result  of  supporting  dynamic  routing.  If  intermediate 
reassembly  was  implemented,  each  gateway  would  then  have  to  delay  each 
incoming  fragment  long  enough  to  determine  tdiether  neighbouring 
fragments  were  taking  the  same  route  and  whether  reassembly  was 
possible.  Reassembly  is  not  guaranteed,  but  the  potential  for 
increasing  congestion  and  adding  to  end-to-end  delay  Is  high. 

A  second  possibility  Is  to  use  the  "Don't  fragment"  option,  relying 
on  the  fact  that  intermediate  networks  arc  required  to  support  segments 
of  at  least  576  octets.  This  Is  a  disguised  form  of  the  previous 
approach,  as  in  order  to  support  it  many  networks  would  have  to 
Implement  a  private  fragmentat ion/ reassembly  protocol  in  the  gatetiays 
adjiicent  to  that  network.  This  may  in  fact  cause  reassembly  to  rake 
place  even  in  situations  where  it  can  give  the  user  no  benefit,  as  the 
packet  has  to  be  fragmented  again  in  the  next  network.  It  does  have  the 
great  advantage,  however,  that  the  two  end  processes  can  efficiently 
adopt  a  common  large  buffer  size  for  bulk  data  transfers. 

4.5  Gateway  to  Local  Network  Flow  Control 

Finally,  we  consider  the  use  that  gateways  can  make  of  services 
available  in  local  netwrks.  For  bulk  data  transfer,  we  wish  the 
gateways  to  make  requests  Indicating  high  throughput  traffic.  As  we  saw 
in  Section  2,  local  nets  typically  do  this  by  allocating  some  proportion 
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of  their  resources  ro  the  requirements  of  the  trsnsfer.  However,  we 
again  come  up  against  problems  due  to  the  datagram  nature  of  the  Catenet 
-  although  the  gateways  may  request  bulk  transfer  facilities  from  a 
local  net,  they  may  not  In  fact  be  able  to  deliver  the  bulk  to  transfer. 
The  ALLOCat Ion  technique  used  In  ARPANET  to  support  file  transfer  (see 
Section  2)  rqlles  on  a  pipe-lining  effect  for  its  efficiency;  It 
delivers  a  new  reservation  with  each  acknowledgement.  If  a  packet  is 
late  reaching  a  gateway,  or  It  enters  the  ARPANET  via  a  different 
gateway  then  this  effect  will  be  lost,  the  reservation  at  the 
destination  node  will  be  cancelled,  and  the  source  node  will  have  to 
reestablish  the  reservation  before  the  next  message  can  be  delivered. 

In  other  networks,  the  effect  of  late  arrival  or  temporary 
rerouting  is  more  likely  to  be  seen  as  a  burst  of  traffic,  possibly 
causing  congestion  problems  Inside  the  networks.  This  sort  of  effect 
will  be  seen  in  satellite  networks  using  reservation  schemes  for  channel 
access.  Since  gateways  are  unable  to  exploit  the  end-to-end 
characteristics  of  the  traffic  to  indicate  the  parameters  which  should 
be  used  to  control  the  channel  algorithm  for  it,  they  can  only  select 
these  parameters  on  the  basis  of  the  packets  immediately  available  in 
the  gateway  queues.  A  gap  in  the  flow  of  traffic  will  cause  loss  of  the 
channel,  which  will  take  at  least  a  quarter  of  a  second  to  recover 
regardless  of  the  reservation  scheme  used.  In  this  time  many  packets 
could  arrive  at  the  ground  station,  and  although  large  reservations  can 
be  made,  they  may  not  be  granted  In  full  if  the  channel  Is  heavily 
loaded.  Thus  it  can  take  some  time  before  the  buildup  is  reduced  to  a 
'normal'  level. 

In  short,  the  DARPA  Catenet  approach  creates  a  large  number  of  flow 
problems  (or  supporting  transnet  bulk  data  transfers  which  are  still 
bring  resolved.  This  Is  primarily  due  to  the  minimal,  datagram  nature 
of  the  Catenet  model.  The  end-to-end  flow  control  techniques  used  by 
TCP,  on  the  other  hand,  are  well  suited  to  file  transfer,  and  may  well 
compensate  for  the  Internal  inadequacies. 


5.  Bulk  Data  Transfer  Across  Public  Data  Networks 
5.1  The  International  Public  Data  Network 

CCITT  Recommendation  X25  [4]  has  laid  down  an  interfacing  standard 
for  the  connection  of  private  data  terminal  equipment  (DTEs)  to  the 
public  data  networks  that  will  become  available  shortly  In  many 
Industrial  countries.  Briefly,  X25  consists  of  three  levels,  defining 
the  physical,  link  level  and  packet  level  Interfaces  between  the  DTE  and 
the  network's  data  communication  equipment  (DCE).  The  packet  level, 
which  concerns  us  here,  defines  procedures  for  the  establishment  and 
breaking  of  virtual  calls  across  the  Interface,  using  a  windowing 
technique  for  flow  control  in  wliich  the  window  size  is  fixed  and 
represents  numbers  of  packets  transferred  rather  than  volute  of  data. 
The  impending  adoption  of  draft  Recommendation  X75  [5],  which  specifies 
a  similar  Interface  to  be  adopted  by  switching  terminal  equipment  (STEs) 
to  connect  adjacent  public  data  networks,  brings  Into  being  the  basis  of 
an  internet  architecture  which  is  of  considerable  Importance  to  many 
data  communication  users.  Since  in  most  countries  (outside  North 


BENNETT:  SUPPORTING  TRANSNET  BULK  DATA  TRANSFER 


b)  The  End-to-end  Virtual  Circuit 
Figure  3t  Data  Transfer  Across  the  Internet lonal  Public  Network 
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AmorUiO  there  will  be  at  most  one  public  data  network,  the  internet 
architecture  can  reasonably  be  called  the  "International"  public  data 
net  v>nrk. 

This  architecture  Is  illustrated  In  Figure  3.  In  It,  a  DTE 
corresponds  to  the  end  host  or  gateway  of  the  DARPA  Catenet  model,  while 
the  STE-to-STE  link  forms  a  particular  type  of  gateway  which  is 
separated  Into  Its  two  local  net  halves.  (By  contrast,  the  single  host 
gateways  Implemented  In  the  DARPA  Catenet  arc  known  as  'monolithic' 
gateways.)  Within  It,  there  are  two  classes  of  transnet  communication  to 
be  considered:  communication  from  DTE  to  DTE  across  more  than  one  public 
network,  but  no  private  networks,  and  communication  between  hosts 
Involving  at  least  one  public  and  one  private  network.  We  shall 
consider  the  first  case  first. 

3.2  Flow  Control  Across  the  International  Public  Netwrk 

As  many  authors  have  pointed  out,  X23  and  X75  arc  merely  Interface 
standards,  which  ensure  the  clean  administrative  separation  o'"  the 
public  network  from  the  user,  on  the  one  hand,  and  from  oth<'t  pub . 'c 
networks  on  the  other.  The  Internal  Implementation  of  the  various 
networks  is  at  the  administration's  discretion,  while  the  end-to-cud 
protocols  wlilch  go  between  the  users  are  by  agreement  between  them. 
Thus  X25  public  data  networks  can  differ  radically  In  their 
architectures:  Telenet  [10]  has  adopted  the  packet  level  of  X23  as  an 
end-to-end  protocol;  Datapac  [29]  has  adopted  It  as  an  access  method  to 
an  internal  datagram  network;  the  Siemen's  proposal  for  public  data 
networks  [26]  will  use  X25  In  a  concatenated  virtual  circuit  fashion. 

These  different  Implementations  will  affect  the  end-to-end 
techniques  used  by  the  application.  Where  an  acknowledgement  from  the 
DCE  Implies  that  the  packet  has  In  fact  reached  the  destination,  as  in 
Telenet,  there  Is  little  need  for  the  user  to  perform  additional  error 
checks,  but  flow  control  becomes  the  user's  responsibility.  However,  if 
it  Is  simply  a  local  acknowledgement  by  the  DCE,  then  the  user  will 
probably  have  to  build  his  own  error  detection  mechanisms  above  X23  In 
addition  to  the  error  recovery  procedures  he  must  build;  on  the  other 
hand,  the  network  may  well  guarantee  to  allocate  buffers  internally  for 
the  connection,  taking  some  of  the  flow  control  responsibility  from  the 
user  . 


All  the  Interfaces  between  networks  and  between  users  and  nettrorks 
are  virtual  c I rcul t -based  for  calls  across  International  data  networks. 
The  Internet  architecture  therefore  presents  the  appearance  of  a  fixed 
path  of  concatenated  virtual  calls,  with  the  routing  algorithm  and 
structure  of  each  alternate  section.  Internal  to  the  network,  undefined. 
The  X75  specification  does  address  the  relationship  of  the  STEs  to  the 
end-to-end  connection  to  some  degree.  Network  RESETS  will  be  propagated 
along  the  end-to-end  path,  and  user  INTERRUPTS  are  specifically  given 
end-to-end  significance.  Windows  between  STEs,  on  the  other  hand,  are 
purely  local.  The  relationship  of  the  STE  to  the  local  net  Is  not 
defined.  It  is  not  strictly  clear,  for  Instance,  whether  an  STE 
attached  to  a  datagram  network  will  transfer  packets  across  the  STE/STE 
Interface  In  the  order  In  which  the  user  created  them  or  In  the  order  In 
which  they  arrive.  If  the  latter  choice  were  made,  the  destination 
process  would  have  to  reserve  buffers  to  handle  misordered  packets,  or 
to  force  end-to-end  retransmissions  should  misordered  packets  arrive. 
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Problems  of  this  nature  arc  fairly  easy  to  overcome  by  makins 
sensible  design  choices;  In  the  example  above,  that  the  STEwtll 
transfer  packets  In  the  user  ordering.  More  subtle  problems  arise  with 
end-to-end  error  control  and  flow  control.  Any  packet  loss  anywhere  In 
the  path  occurring,  for  Instance,  because  of  a  RESET  between  two  STEs, 
will  be  propagated  down  the  rest  of  the  path.  The  packer  loss  will 

merely  be  compounded  by  the  requirement  of  X75  that  the  RESET  should 
also  be  propagated  do«m  the  rest  of  the  path.  Losses  occurring  for 

other  reasons  may  be  propagated  to  the  end  of  the  path  without 
detection. 

Most  importantly,  unless  X25  acknowledgements  have  purely  local 
significance  in  the  OTE-DCE  Interface,  their  significance  across  more 
than  one  public  net  is  completely  indeterminate:  they  may  represent 
acceptai\ce  by  the  first  STE,  by  the  remote  DCE  or  DTE,  or  by  any 
intermediate  STE.  Hence  the  user  cannot  regard  International  calls  as 
having  anything  other  than  local  flow  control.  Because  of  the  potential 
differences  In  flow  control  strategies  within  public  nets,  the  user 
cannot  make  any  assumptions  about  the  buffer  reservations  that  will  be 
made  for  his  connection  outside  his  local  public  network. 

Clearly,  this  situation  has  to  be  remedied  by  the  introduction  of 
an  end-to-end  protocol  across  the  public  data  system.  It  has  been 
suggested  (16]  that  the  end-to-end  protocol  should  be  Implemented 
between  source  and  destination  DCEs.  It  Is  highly  unlikely  that  this 
solution  will  be  adopted,  as  it  requires  all  public  data  networks  to 
adopt  the  same  end-to-end  protocol,  which  will  compromise  the 
independence  of  the  administrations  to  modify  their  internal 
architecture  at  will.  It  is  far  more  likely  that  appropriate  DTE-to-DTE 
protocols  will  have  to  be  Implemented  by  the  groups  of  users  %dio  wish  to 
communicate. 

5.3  Flow  Control  Across  Constituent  Public  Networks 

Given  that  a  global  architectural  framework  has  to  be  imposed  above 
the  public  data  networks,  to  provide  end-to-end  reliability,  ordering, 
and  some  flow  control  technique,  we  can  now  consider  how  well  individual 
public  data  nets  can  support  transnetwork  bulk  data  transfers.  X25  and 
X73  provide  several  user  facilities  which  are  of  great  assistance  here, 
but  they  also  have  several  defects  idiich  arise  from  the  lack  of 
specification  of  the  end-to-end  significance  of  the  architecture. 
Window  sizes  are  only  defined  across  the  Interface,  and  there  Is  no 
guarantee  that  a  request  to  reserve  large  buffers  will  be  carried  from 
DCE  to  STE  or  from  STE  to  STE;  however,  STEs  are  required  to  transmit 
user  throughput  class  requests  unchanged.  The  significance  of  these  can 

vary  from  net  to  net:  one  net  can  assume  that  they  represent  a 

guaranteed  rate,  the  next  that  they  represent  a  maximimi  rate  and  the 

next  that  they  represent  a  minimum  rate.  In  any  case,  they  only 

represent  throughput  across  the  Interface,  and  cannot  be  used  as  a  basis 
for  guaranteeing  high  throughput  across  s  local  public  network. 

Nevertheless,  user  control  of  flow  rates  between  STEs  scross  local 
networks  Is  greater  than  It  Is  between  gateways  in  the  DARPA  Catenet.  A 
means  of  requesting  high  bandwidth  services  does  exist,  and  because  of 
the  concatenated  virtual  circuit  nature  of  the  path,  one  can  guarantee 
that  all  the  packets  Involved  in  the  transfer  will  take  that  path. 
Public  data  nets  relying  on  short-term  pipelining  effects  of  the  sort 
mentioned  In  Section  4  will  have  a  greater  chance  of  keeping  the 
pipeline  full  than  the  DARPA  Catenet  does. 
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X23  and  X75  provide  a  "More  Data"  option  to  Indicate  a  sequence  of 
related  packets.  X2S  also  supports  a  number  of  possible  packet  sices, 
all  powers  of  2  (with  the  exceptional  case  of  253  octet  packets).  Thus 
It  Is  possible  for  an  X23  network  to  use  the  "More  Data"  flag  to 
assemble  small  packets  Into  larger  ones  Inside  the  network,  a  feature 
%jhlch  would  promote  packet  efficiency  and  reduce  the  reassembly  overhead 
at  the  destination.  X73,  however,  only  supports  a  maxlmiaa  packet  sice 
of  128  octets,  which  Is  the  size  guaranteed  by  all  X23  administrations. 
Intermediate  reassembly  Is  therefore  only  a  reasonable  option  at  the 
destination  netwrk,  but  by  the  same  logic,  fragmentation  Is  only 
necessary  In  the  source  network  (or  the  first  STE  encountered). 

3.4  Flow  Control  Between  Private  and  Public  Data  Netwrks 

The  connection  of  private  data  networks  to  public  data  nettiorks  Is 
less  well  defined.  The  DTE  %ihlch  acts  as  a  gateway  between  the  tw 
networks  clearly  supports  X23  virtual  circuits  across  the  DTE/DCE 
Interface,  and  this  tends  to  encourage  the  extension  of  the  concatenated 
virtual  circuit  model  to  cover  the  whole  domain  over  which  the 
end-to-end  transport  protocol  operates.  A  datagram  structure  betwen 
the  source  host  and  the  gateway  will  produce  all  the  additional 
difficulties  on  flow  control  that  we  have  noted  in  Section  4.  Moreover, 
if  the  network  Is  connected  to  the  public  network  at  more  than  one  point 
and  it  allows  dynamically  routed  datagrams  to  enter  the  public  net 
through  any  of  them,  then  all  the  potential  advantages  %ihich  the  virtual 
circuit  structure  of  the  public  networks  can  provide  are  lost. 

This  argunent  applies  regardless  of  the  end-to-end  context  within 
which  the  transfer  Is  taking  place.  If  an  end-to-end  transport  protocol 
is  in  operation  it  will  perform  the  necessary  end-to-end  controls;  If 
not,  the  transfer  can  be  broken  down  into  Independent  stages.  In  either 
case,  a  concatenated  virtual  circuit  structure  offers  the  best  way  for  a 
private  network  to  make  use  of  all  available  facilities  for  maximising 
throughput  across  the  international  public  network,  despite  the  costs  of 
maintaining  connection  data. 


6.  An  Experimental  Transnet  File  Transfer  System 

It  is  clear  from  the  above  discussion  that  many  of  the 
architectural  issues  related  to  flow  control  for  transnet  services  are 
very  open.  In  particular,  there  is  a  real  need  to  study  the  integration 
of  flow  control  techniques  at  various  levels  within  the  transnet  system, 
both  to  extract  the  maximum  possible  efficiency  from  the  system  and  to 
overcome  potentially  harmful  Interactions  which  can  occur  without  such 
integration.  The  section  describes  an  experimental  framework  being 
developed  to  do  this. 

University  College  London  has  been  Involved  In  the  study  of 
networks  for  a  number  of  years,  notably  In  connection  with  the  ARPANET, 
SATNET,  and  the  experimental  UK  Post  Office  Network  EPSS  [23].  Over  the 
next  two  years,  the  UCL  configuration  will  undergo  some  considerable 
changes,  ns  the  direct  ARPANET  link  Is  replaced  by  a  transnetwork  link 
through  SATNET  using  TCP,  EPSS  Is  phased  out,  and  we  increase  our 
Involvement  with  a  number  of  X23-based  packet-swl tched  networks.  A 
simplified  version  of  the  new  configuration  Is  given  in  Figure  4.  The 
central  component  of  the  new  configuration  is  an  LSI-11  which  acts  as  a 
transport-level  protocol  convertor  between  the  DARPA  Catenet  on  the  one 
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Figure  2i  DARPA  Catenet  Architecture 
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Flic  transfer  service  can  be  provided  In  tw  ways:  either  as  an 
end-to-end  service,  between  hosts  in,  say,  the  ARPANET  and  the  public 
network,  or  as  a  staged  service  with  two  components,  one  over  the  DARPA 
Catenet,  one  over  the  public  data  network.  The  second  option  can  be 
done  in  two  ways  -  either  as  a  batch  service,  in  which  the  whole  file  Is 
t r.'.nsferred  to  a  disc  on  the  gateway  and  then  transferred  again  to  the 
second  network,  or  as  a  direct  mapping  between  t%K>  file  transfer 
protocols  at  the  gateways. 

Neither  of  staging  options  offers  a  particularly  attractive 
solution.  The  batch  staging  solution  requires  that  every  gateway  must 
maintain  a  large  secondary  storage  system  to  store  all  the  intermediate 
files  that  may  be  sent  through  It,  which  is  very  inefficient,  expensive 
for  the  gateway's  administration  and  time-consuming  for  the  user. 
Moreover,  its  costs  rise  linearly  with  the  nuaber  of  networks  traversed, 
so  even  if  it  is  acceptable  for  two  net%«orks  it  rapidly  becomes  an 
unacceptable  approach.  Problems  also  arise  with  the  protocol-mapping 
solution.  Such  mappings  arc  unlikely  to  be  exact,  and  the  result  will 
be  an  end-to-end  service  which  does  not  offer  all  the  features  of  either 
half,  and  has  no  end-to-end  controls  built  in  for  reliability  or  flow 
control.  Both  solutions  require  that  gateways  implement  at  least  two 
complete  file  transfer  protocols  and  the  additional  software  needed  for 
the  staging. 

Ideally,  then,  one  would  like  to  offer  an  end-to-end  transnet  file 
transfer  service  directly.  One  of  the  more  important  recent 
developments  in  the  field  of  protocol  design  has  been  the  development  of 
net  work- independent  protocols,  that  is,  protocols  which  make  a  minimum 
number  of  assumptions  about  the  nature  of  the  underlying  service.  TCP 
was  one  early  example  of  this  class  of  protocols.  A  network-independent 
file  transfer  protocol  has  been  proposed  by  the  EPSS  Higher  Level 
Protocols  Croup,  which  for  this  reason  will  be  referred  to  as  the  NI  FTP 
[17].  This  protocol  defines  a  "conceptual  file  store"  and  a  means  of 
transferring  files  between  two  sides  based  on  this  conceptual  file 
store.  The  protocol  only  requires  two  services  of  the  underlying 
medium:  that  it  provide  sequenced  delivery  with  an  acceptably  low  error 
rate,  and  that  there  exists  some  means  of  indicating  a  "transport 
service  reset"  to  the  transfer  program  in  the  event  of  serious  trouble 
in  the  medium.  This  proposal  is  being  implemented  at  a  number  of  sites 
in  the  UK,  mostly  on  EPSS,  and  is  being  circulated  widely  in 
international  networking  groups. 

Tite  protocol  uses  a  scheme  of  sequenced  file  recovery  marks  and 
provides  a  means  of  specifying  a  fixed,  mark-based  window.  This  gives  a 
means  of  end-to-end  flow  control  and  error  control  which  Is  intended  to 
reflect  the  characteristics  of  the  end  storage  devices:  for  instance,  a 
file  being  read  in  from  a  card  reader  might  be  associated  with  a  mark  on 
every  card,  and  have  a  window  of  one  card,  as  the  card  reader  cannot  be 
backed  up;  a  file  transferred  between  two  discs  would  only  insert  file 
markers  rarely,  and  might  use  the  window  to  match  the  access  rates  of 
the  two  discs.  The  error-control  facility  of  restarting  the  transfer 
from  the  last  acknowledged  mark  is  Intended  to  be  used  in  the  case  of  a 
transport  station  reset,  and  so  does  have  some  impact  as  an  end-to-end 
error  recovery  scheme  for  the  network.  The  window  scheme  also  will  have 
some  effect  on  the  flow  across  the  network;  Indeed,  with  transfers 
between  similar  devices  across  a  variable  medius,  its  principal  effect 
will  be  to  match  the  storage  device  and  the  medius. 
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The  major  difference  between  this  file  transfer  configuration  and 
the  ones  discussed  In  the  previous  sections  Is  that  here  there  is  no 
end-to-end  transport  protocol.  One  can,  however,  define  a  complete  path 
of  concatenated  virtual  circuits,  with  final  reliability  control 
residing  In  the  file  transfer  protocol  Itself.  On  the  SATNET  side  of 
the  protocol  convertor  end-to-end  transport  Is  provided  by  TCP.  It  is, 
of  course,  also  possible  to  build  the  N1  FTP  above  the  standard  ARPANET 
NCP,  In  which  case  another  connection  would  be  set  up  between  a  host 
running  both  TCP  and  NCP  somewhere  on  the  ARPANET  and  the  terminating 
host.  The  transport  protocol  across  the  X25-bascd  net  is  at  present 
undecided,  but  when  it  is  determined  it  will  be  Incorporated  into  the 
protocol  convertor  residing  above  X25.  The  protocol  layer  structure  of 
the  protocol  convertor  Is  shown  In  Figure  4.b. 

The  final  system  bears  a  strong  resemblance  to  the  architectural 
models  we  have  developed  for  other  transnet  systems,  especially  that  of 
Figure' l.b.  Here,  by  analogy,  we  are  defining  a  'meta-net'  within  an 
end-to-end  context  provided  by  the  NI  FTP,  and  the  'local  networks'  are 
the  DARPA  Catenet,  the  public  data  networks,  that  part  of  ARPANET 
restricted  to  NCP  communication,  etc  -  I.e.  they  are  based  on  a 
division  by  transport  protocol  rather  than  by  physical  network 
structure.  To  carry  the  analogy  further,  the  meta-gateways  of  this 
system  are  the  hosts  which  run  different  transport  protocols  within  the 
framework  of  the  'meta-net'.  Within  the  meta-gateways,  above  the 
transport  services,  the  central  module  supports  the  specific 
requirements  of  the  end-to-end  application  protocol  defining  the  domain 
of  the  'meta-net'. 

The  functions  of  this  central  module  are  in  many  respects  similar 
to  those  of  ordinary  gateways.  For  Instance,  it  must  support  similar 
addressing  and  routing  functions,  and  it  must  set  up  and  close  down 
meta-gateway  to  meta-gateway  virtual  circuits,  and  support  the  opening 
and  closing  of  the  end-to-end  connection.  Its  primary  function  Is  to 
support  hlgh-bandwidth  end-to-end  throughput  rates  by  maintaining 
sequencing,  by  supporting  and  matching  appropriate  local  flow  control 
and  congestion  control  strategies,  by  using  locally  available  options 
for  high-bandwidth  flow,  by  tailoring  the  size  of  the  data  units  to  be 
as  efficient  as  possible  for  the  next  transport  section,  and  by 
indicating  a  “transport  service  reset"  to  the  file  transfer  processes  in 
the  event  of  net%a)rk  failure. 

The  techniques  which  might  be  used  to  do  this  are  the  primary 
object  of  the  investigation.  Some  Initial  suggestions  arc  made  here  to 
Indicate  the  nature  of  possible  solutions.  End-to-end  control  will  be 
based  on  the  file  transfer  window  Itself.  At  least  In  the  early  stages 
of  the  Investigation,  it  will  be  assumed  that  transfers  are 
disc-to-disc;  hopefully.  It  will  be  shown  that  it  will  be  possible  to 
relax  this  restriction  and  still  maintain  adequate  end-to-end  flow. 
Over  the  DARPA  Catenet,  large  window  techniques  would  be  adopted,  based 
probably  on  conservative  strategies  for  buffering  and  back-off 
retransmission  strategies  for  congestion  control.  Similar  strategies, 
appropriately  modified,  may  be  adopted  over  X25.  Packers  coming  in  from 
the  X25  side  may  well  be  reassembled  into  TCP  packets  which  have  a  size 
related  to  the  record  size  of  the  file;  in  the  other  direction,  they 
will  be  fragmented  into  128  octet  units,  wlitch  contain  exactly  2  file 
subrecords.  Over  X23  a  high  throughput  class  will  be  selected,  and  the 
unity  of  record  fragments  will  be  Indicated  by  use  of  the  "More  Data" 
facllliy.  User  options  in  the  DARPA  Catenet  present  a  considerable 
problem:  as  we  have  seen,  none  of  those  currently  proposed  are  without 
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undesirable  side  effects.  Probably,  none  will  be  adopted  until  the 


position  Is  clarified.  It  Is 
packet  sizes,  combined  with  the 
subrecord  sizes  and  X2S  packet 
fragment  at  I  on/ reassembly  policy. 


expected  that  the  flexibility  of  TCP 
close  relationship  between  NT  FTP 
sizes  should  lead  to  an  optimal 


7.  Conclusions 

Bulk  data  transfer  is  a  network  service  which  demands  specialised 
support  from  packer  switched  networks  in  order  to  give  efficient 
service.  When  the  transfer  is  taking  place  across  more  than  one 
network,  the  problems  are  compounded  by  the  heterogeneity  of  the 
communicat ions  medlixn  thereby  created.  Neither  of  the  transnetwork 
architectures  examined  In  this  paper  give  entirely  adequate  support.  It 
is  instructive  to  examine  the  reasons  for  this.  In  the  DARPA  Catenet, 
we  have  an  end-to-end  transport  mechanism  which  would  seem  to  be  ideal 
for  bulk  data  transfer.  The  problems  we  encountered  arise  because  of 
the  minimal  assumptions  made  about  the  underlying  medltaa.  Again  and 
again  we  found  that  the  fact  that  we  can  only  assume  a  basic  datagram 
service  in  the  Catenet  interferes  with  the  effectiveness  with  which  we 
can  support  the  end-to-end  service.  Most  of  the  problems  could  be 
greatly  simplified  if  a  virtual  call  service  existed  within  the  Catenet 
framework.  From  the  standpoint  of  technical  efficiency,  the  case  for  a 
virtual  circuit  option  in  the  DARPA  Catenet  is  strong.  The  major 
obbstacle  to  Its  implementation  is  the  difficulty  of  making  it 
compatible  with  adaptive  routing. 

The  source  of  the  deficiencies  in  public  networks  for  supporting 
file  transfer  is  rather  different.  Here  we  have  the  basic  virtual 
circuit  structure  required,  and  we  have  some  service  facilities  which 
are  adequate  for  our  purpose.  The  main  problems  are  due  to  a  deficiency 
which  has  often  been  noticed  in  the  X23  debate:  the  lack  of  a  global 
architectural  framework.  This  makes  It  difficult  to  Judge  the  effects 
of  actions  across  X25  and  X75  Interfaces,  as  their  end-to-end  effects 
arc  Indeterminate.  The  current  version  of  X75  shows  some  awareness  of 
this  criticism,  but  flow  control  is  one  area  in  which  no  clear  answers 
have  emerged. 

The  two  global  architectures  examined  here  differ  In  many  respects. 
The  experiment  outlined  In  section  6  is  an  attempt  both  to  find  flow 
control  policies  suitable  for  bulk  transfer  In  each,  and  to  examine 
techniques  for  marrying  the  two  to  provide  an  efficient  end-to-end 
serv  Ice . 
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ir  generacea  rnc  aessage.  me  opruaisric  policy  oases  window  size 
calculations  on  some  predictive  heuristic  to  give  a  reasonable  estiaate 
of  what  the  receiving  TCP  expects  to  possess  In  the  near  future.  As  one 
would  expect,  slaulatlon  studies  (11]  have  shown  that  the  conservative 
policy  Is  the  best  one  to  follow,  especially  when  buffer  space 
aanageaent  is  the  responsibility  of  the  user  process,  which  is  the 
policy  encouraged  by  the  TCP  iapleaentors.  In  this  case,  an  optlaistlc 
policy  must  try  to  gauge  the  deaands  that  the  user  process  is  going  to 
aake  on  the  TCP  and  the  resources  that  it  will  aake  available.  The 
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4.5  The  EPSS-ARPANET-Satnet  Demonstrations 


In  Fig.  2.1  we  described  the  UCL  INDRA  configuration.  We 
have  two  lines,  at  48  and  2.4  Kbps,  into  EPSS.  The  EPSS  is 
a  Virtual  Call  network,  with  a  well-defined  set  of  protocols 
for  Network  Access,  Transport  Station,  Virtual  Terminals 
and  File  Transfer.  None  of  the  protocols  are  very  similar 
to  the  ARPANET  ones,  though  reasonable  mappings  are  possible, 
e.g.  between  TELNET  on  ARPANET  and  a  Virtual  Terminal 
Protocol  (VTP)  on  EPSS.  UCL  has  set  up  a  mapping  gateway, 
so  that  Terminal  streams  over  EPSS  can  be  sent  over  ARPANET, 
This  system  is  described  in  some  detail  elsewhere  (23). 

In  the  documentation  on  the  INDRA  software  systems  Gnome  (24) 
and  Santa  (25),  we  showed  how  terminals  attached  to  an 
ARPANET  TIP  at  UCL  could  be  used  to  control  Data  Generators 
sited  in  the  ARPANET-SATNET  gateways.  A  TENEX  on  ARPANET 
was  used  to  control  the  experiments;  communication  with  the 
Controller  was  via  terminal  interaction  over  ARPANET. 

In  June  1978,  a  large  demonstration  was  organised  to  show 
the  capabilities  of  EPSS.  As  part  of  this  demonstration, 
the  capabilities  of  the  two  previous  paragraphs  were  combined 
to  allow  Satnet  measurements  to  be  performed  by  a  concaten¬ 
ation  of  SATNET,  ARPANET  and  EPSS.  The  forms  of  connection 
used  were  chosen  for  pragmatic  reasons;  they  required 
extensive  protocol  conversions  at  several  points.  Thus,  the 
demonstrations  do  not  present  a  reasoned  approach  to  how 
Internet  Services  should  be  provided  in  the  future.  However, 
a  very  effective  Internet  demonstration  ising  all  three 
networks  was  set  up,  which  worked  well. 

The  physical  topology  of  the  EPSS-ARPANET-Satnet  demonstra¬ 
tion  is  shown  in  Fig.  4.1.  The  relevant  portions  of  ARPANET 
are  the  IldPs  at  BBN  and  ISI  to  which  were  attached  the  Satnet 
Monitor  Center  TENEX  (SMC)  and  the  TENEX  (EC)  running  the 
Santa  Controller.  While  the  TENEX  or  TOPS  20  running  the 
Data  Acquisition  software  could  be  a  different  machine,  we 
usually  used  EC  also  for  this  function.  To  it  Norsar,  BBN 
and  UCL  TIPs  (or  IMPs)  were  attached  to  PDP  11/34  Gateways. 
These  Gateways  were  Hosts  on  both  ARPANET  and  EPSS.  For 
convenience,  character  terminals  were  attached  to  the 
UCL  TIP. 

For  data  processing  between  ARPANET  and  EPSS,  the  UCL  PDP9 
did  protocol  mapping  of  the  End-to-End  Transport  Protocols 
and  the  Virtual  Terminal  Protocols.  The  mapping  was  based 
on  concatenated  virtual  calls.  No  Internet  functions  (like 
the  Internet  packet  format)  were  used.  The  data  passing 
between  the  Experiment  Controller  Santa  and  the  Gnome 
traffic  generators  in  the  Gateways  were  raw  ARPANET  packets. 
The  Gnome  traffic  generators  produced  packets  with  the  correct 
internet  leaders  to  pass  through  the  ARPANET-Satnet  gateways 
to  their  data  Acquisition  counterparts  in  the  destination 
gateway(s). 
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Chiiractt'r  terminal  traffic  originated  both  from  terminals 
attached  to  the  UCl.  TIP,  and  from  terminals  directly  dialling 
into  KPSS.  Since  all  UCL  character  terminal  multiplexing 
is  performed  by  the  UCL  TIP,  the  communication  path  between 
the  UCL  terminals,  the  Santa  Controller  and  the  Satnet 
Monitoring  Centre  were  complex.  All  terminal  traffic  passed 
from  the  TIP  (T  in  Fig.  4.1)  with  protocol  conversion  via 
P2  and  EPSS  over  the  2.4  Kbps  line.  Both  the  traffic  from 
I  lie  UCL  terminals  and  the  others  then  passed  through  EPSS 
and  the  48  Kbps  line  through  PI  (with  EPSS-ARPANET  protocol 
conversion)  to  the  TIP.  From  there  a  normal  ARPANET  connec¬ 
tion  was  made  to  the  EC  and  SMC;  the  terminal  dialog  between 
UCL  and  EC  resulted  in  control  traffic  going  between  EC 
and  Gnome  in  the  controlling  Gateways  CG. 

This  traffic  itself  generated  control  information  between 
Gnome  in  GC  and  tliat  in  other  gateways  G  through  Satnet. 

The  Controllers  in  the  G  generated  experimental  traffic 
over  Satnet  which  produced  statistics  data  in  the  Gnomes 
in  G.  Tliis  statistics  data  was  passed  through  ARPANET  to 
the  Experimental  Acquisition  system  (EA  usually  the  same 
as  EC).  Finally,  the  statistics  were  partially  pre- 
processed  in  EA,  and  displayed  on  the  UCL  terminals  via 
ARI'‘ANET.  They  were  also  sent  via  the  UCL  TIP  and  PI  through 
ARI’ANET  and  EPSS  with  the  appropriate  protocol  conversion, 
to  the  RL.  Here  an  IBM  .360/195  analysed  fully  the  data 
and  produced  output  on  an  electron  beam  recorder.  This 
whole  process,  while  convoluted,  worked  reliably  -  and 
demonstrated  the  multi-net  capability  already  in  existence. 


Fiff  4*1  Schematic  of  Coupled  Networks 


V, 
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5.1  Introduction 


In  principle,  the  measurement  techniques  were  developed 
in  previous  years,  and  were  discussed  in  the  last  annual 
report.  During;  1978  there  have  been  several  new  releases 
of  SIMP  software,  incorporating  new  packet  structures. 

The  Gnome  traffic  Generators,  and  the  Santa  Controllers, 
have  had  to  keep  pace  with  the  changes.  These  uncovered 
a  number  of  service  problems. 

One  problem  was  the  synchronisation.  Because  we  had  to 
timestamp  individual  packets  it  was  essential  either  to 
devise  methods  of  calibrating  the  clocks  in  the  different 
computers,  or  to  remove  the  need  to  compare  the  times 
at  different  sites.  Calibration  was  accompanied  by 
specific  calibration  packets.  By  use  of  the  round-trip 
packets,  the  need  for  many  timing  calibrations  was 
avoided.  Finally,  by  repeating  the  calibration  at 
intervals,  and  performing  linear  interpolation  between 
the  intervals,  significant  drift  could  be  cancelled  out. 

Once  significant  numbers  of  experiments  were  performed, 
it  became  essential  to  record  the  measurement  statistics 
in  a  structural  data  base.  This  data  base  is  held  on  the 
RL  360/195.  A  detailed  description  both  of  the  data  base 
software  and  of  how  it  is  used  is  given  elsewhere  (26). 

For  the  first  time  we  derived  very  valuable  measurement 
results  from  the  SATNET  experiments.  These  experiments 
and  their  results  are  described  briefly  in  (27).  Because 
the  SATNET  measurement  project  terminated  in  the  year, 
rather  more  detail  will  be  given  here  than  for  other 
projects . 


5 . 2  Bata  Analysis  Tools 

The  first  part  of  1978  was  used  to  improve  the  data 
analysis  tools.  Up  to  this  point  much  time  had  been 
spent  on  the  mt'asuremi'nt  mechanism  whicli  involved  the 
traffic  generation  and  experiment  control  software. 

The  system  was  initially  implemented  so  that  the  machine 
that  controlled  the  experiment  also  collected  the  data. 

As  the  controlling  machine  was  at  ISI  (Los  Angeles)  in 
the  US,  the  data  at  the  end  of  the  experiment  was  collected 
there  too. 

Even  though  the  data  was  collected  in  the  US,  a  decision 
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was  taken  to  perform  all  the  analysis  on  the  IBM  370/195 
at  Rutherford  Laboratory  in  Oxfordshire.  Rutherford  was 
selected  as  it  not  only  had  plenty  of  computer  power 
but  also  had  an  impressive  set  of  graphics  hardware. 

The  main  graphics  machine  was  the  FR80  electron  beam 
recorder,  a  machine  that  enabled  many  different  forms 
of  graphics  output  to  be  produced,  from  large  black 
on  white  paper  hardcopy  to  the  production  of  35mm  colour 
slides.  We  found  the  ability  to  produce  slides  of  our 
results  an  extremely  important  asset. 

At  the  end  of  each  experiment,  the  data  was  transferred 
from  the  ARPANET  where  it  had  been  collected  to  the 
Rutherford  Lab  via  the  London  Gateway  at  University 
College.  This  operation  was  usually  completed  within 
30  minutes  of  the  end  of  an  experiment,  although  the 
actual  time  depended  upon  the  amount  of  data  to  be 
transferred.  (The  data  could  be  transferred  at  the 
rate  of  1000  bits  per  second). 

At  Rutherford  the  data  was  analysed  by  a  suite  of  programs 
and  placed  on  a  database  that  was  designed  and  created 
especially  for  the  purpose  of  collecting  these  statistics. 
The  database  had  the  ability  to  collect  approximately 
30  experiments  online  at  a  time.  After  the  database  had 
been  filled  the  older  experiments  could  be  archived  on  to 
magnetic  tape.  Such  archived  experiments  could  be  rapidly 
restored  onto  the  database  if  the  need  arises. 


The  graphics  programs  were  written  to  access  this  database 
and  extract  experimental  information.  These  routines 
would  collect  data,  according  to  user-specified  parameters, 
and  display  the  results  in  graphics  form.  These  routines 
had  the  ability  to  display  a  crude  drawing  on  the  line 
printer  or  to  produce  rather  better  graphics  on  the 
Computek  display  unit  that  we  have  on  site  in  London. 

The  experimenter  could  then  route  the  graphics  output 
to  the  FR80  and  the  colour  slides  would  be  received 
about  a  week  later.  The  database  access  routines  also  had 
the  ability  to  compare  the  results  of  different  experiments, 
and  even  to  use  the  information  of  a  complete  set  of 
experiments  to  deduce  trends  in  the  data. 
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The  actual  experiments  began  early  in  the  year,  but  at 
first  these  were  just  trial  experiments  to  prove  that 
that  system  was  working  correctly. 


Initially  there  were  only  three  types  of  traffic 
generation  possible:  Stream  Traffic,  Bulk  Traffic  and 
Interactive  Traffic,  Stream  Traffic  was  found  to  be 
important  for  the  measurement  of  the  network  capabilities 
as  this  method  enabled  messages  to  be  generated  at 
fixed  time  intervals.  As  the  inter-message  time  at  the 
transmit  side  was  constant  the  inter-message  time  at 
the  destination  could  be  measured  to  see  how  it  differed 
from  the  source  side.  The  disturbance  introduced  into 
the  traffic  stream  could  then  be  measured  directly. 

As  the  frequency  of  packets  was  increased  it  was  also 
possible  to  see  how  the  network  reacted  to  the  heavier 
traffic  load. 


The  Bulk  Traffic  mode  was  changed  shortly  after  the 
first  set  of  experiments  to  enable  it.  to  simulate  a 
windowing  technique.  This  transmission  method  meant 
that  there  was  a  "window"  of  messages  at  the  source 
such  that  only  this  window  size  was  permitted  to  be 
outstanding  in  the  network  at  any  one  time.  Transmission 
at  the  source  is  then  frozen  until  an  acknowledgement 
is  received  for  at  least  part  of  the  data.  This 
technique  was  based  heavily  upon  TCP  concepts  and  used 
parts  of  the  same  algorithm. 


It  is  interesting  to  note  that  the  interactive  traffic 
type  was  never  found  to  be  very  useful.  The  interactive 
traffic  was  tested  on  a  few  occasions  but  the  results 
were  found  to  be  difficult  to  analyse  due  to  the  number 
of  variables  involved.  The  stream  traffic  type  and  the 
bulk  traffic  type  were  found  to  achieve  far  more  in  terms 
of  network  measurement.  The  interactive  traffic  was 
based  upon  messages  being  sent  back  from  the  destination 
each  time  a  message  was  found  that  the  response  of  the 
network  could  be  measured  more  accurately  with  the 
stream  traffic  type. 


5 . 4  Measurement  Limitations 

Unfortunately  the  generation  machine  was  found  to  be 
too  slow  to  explore  fully  the  data  capacities  of 
SATNET,  The  machines  could  not  generate  traffic  faster 
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than  8  packets  per  second,  far  below  what  the  network 
could  be  expected  to  accept.  This  was  attributed  to 
the  operating  system  which  although  very  good  from  a 
debugging  point  of  view  was  very  slow  in  terms  of 
processing  packets  from  a  user  process  to  the  network 
or  vice  versa.  Eventually  we  found  that  we  had  to  live 
with  this  limit  of  eight  packets  per  second  and 
concentrate  instead  on  the  disturbance  and  delays 
occurring  in  this  traffic  rather  than  pushing  a  high 
data  volume  through  the  network. 

Our  techniques  permitted  the  detection  of  at  least  one 
"bug"  in  the  network  software.  We  found  that  even  at 
low  data  rates  the  packets  would  arrive  wildly  out  of 
sequence  at  the  destination.  This  was  later  found  to 
be  a  bug  in  the  satellite  IMPs  whereby  the  test  to 
place  the  packets  in  the  output  queue  had  been  reversed 
so  that  the  packets  were  placed  at  the  front  of  the 
queue  when  they  should  have  been  at  the  end. 


5 . 5  Measurement  Results 

The  experiments  were  perfonned  in  several  phases. 

The  first  set  of  experiments  consisted  of  echoing 
traffic  from  different  points  in  SATNET.  This  echo 
loop  was  started  as  an  internal  loop  within  the  source 
machine  and  then  expanded  until  it  looped  traffic  to 
the  local  SIMP,  then  to  the  satellite,  the  destination 
SIMP  and  finally  the  destination  machine.  These  experiments 
allowed  the  behaviour  of  the  network  to  be  studied  in 
some  detail  and  the  bottlenecks  of  the  system  to  be 
determined. 

Essentially,  we  found  that  the  bottleneck  occurred  in 
the  source  machine.  The  maximum  packet  rate  internally 
looped  within  this  machine  was  just  over  12  messages 
per  second.  (Compare  this  to  another  ARPANET  experiment 
that  had  a  microprocessor  handling  200  messages  per 
second).  When  the  loop  was  extended  to  the  SIMP  so  that 
the  traffic  travelled  along  the  VDH  line  and  back  again 
the  throughput  dropped  to  10  messages  per  second.  Thus 
the  extra  overhead  of  transmitting  the  packets  to  the 
SIMP  and  back  reduced  the  throughput  only  slightly. 

For  most  traffic  across  the  network  we  measured  a  delay 
of  between  1.1  and  1.75  seconds,  with  a  peak  at  1.4 
seconds.  This  was  measured  with  packets  being  sent  every 
140mi 1 liseconds ,  but  when  this  frequency  was  increased 
the  delays  went  up  dramatically,  mainly  due 
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to  the  queueing  in  the  source  machine  being  unable  to 
handle  them. 

The  windowing  experiments  achieved  some  very  interesting 
results.  Here  we  measured  the  time  from  sending  a  packet 
to  the  time  a  reply  ( acknowlegement )  was  received  and 
found  that  this  varied  with  the  size  of  the  window. 

The  best  round-trip  time  of  2  to  3  seconds  was  achieved 
with  a  window  size  of  6  messages;  when  the  window  was 
expanded  to  15  messages  the  maximum  delay  rose  to  4 
seconds  for  the  round-trip.  In  fact  once  the  transmission 
rate  at  the  source  exceeded  the  throughput  that  the  network 
could  take  the  message  delays  increased  until  eventually 
all  the  buffer  space  was  exhausted.  Thus  we  saw  that 
opening  up  the  window  larger  than  the  throughput  rate 
of  the  network  not  only  does  not  improve  the  throughput  but 
is  detrimental  to  the  other  users  of  the  network.  Once  the 
window  size  gets  too  large  the  messages  sit  in  buffers 
waiting  for  transmission  and  thus  prevent  other  tasks  from 
using  the  buffer  resource. 


Conclusions 


The  measurement  results  more  than  justified  the  time 
spent  in  the  development  of  the  measurement  tools.  The 
system  we  developed  was  able  to  measure  many  aspects  of 
computer  network  performance.  The  perforraace  of  individual 
machines  could  also  be  determined.  Even  more  impressive 
was  the  fact  that  these  measurements  could  be  run  and 
co-ordinated  in  machines  that  were  many  thousands  of 
miles  apart  in  different  continents.  Within  minutes  of 
an  experiment  terminating  we  could  have  the  first  results 
displayed  on  our  screens.  This  rapid  access  to  results 
and  the  ease  with  which  complex  experiments  could  be  run 
enabled  us  to  pursue  an  extensive  set  of  experiments  in 
a  relatively  short  time. 

Perhaps  even  more  important  than  the  results  themselves 
was  the  experience  we  gained  in  timestamping,  the  foundation 
of  our  measurement  effort.  After  several  months  of 
measurements  we  built  up  an  enormous  amount  of  experience 
in  the  use  of  timestamps.  We  had  to  solve  various 
problems  associated  not  only  with  the  mechanics  of 
collecting  timestamps  but  also  of  varying  clock  rates 
between  machines  operating  at  great  distances.  We 
approached  the  problem  of  global  clock  synchronisation 
and  attempted  to  find  solutions.  We  then  moved  on  to 
develop  measurement  methods  that  did  not  require  clocks 
to  be  synchronised,  and  yet  the  timestamps  were  still 
able  to  yield  detailed  measurement  results. 


VI.  FACSIMILE  ACTIVITIES 
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The  actual  grant  under  which  our  facsimile  work  has  been 
carried  out  expired  in  July.  A  final  report  on  that 
project  has  been  written  (28). 

The  previous  annual  report  included  two  papers  which  summ¬ 
arised  much  of  our  work.  These  papers  were  somewhat  revised 
before  publication  (29,30).  Another  paper  mentioned  in 
the  previous  report,  on  terminals  for  facsimile  has  also 
appeared  since  that  date  (31).  These  papers  are  only  men¬ 
tioned  because  the  previous  reference  list  in  (1)  was  inconplete.  A 
much  more  significant  extension  and  revision,  with  in- 
depth  detail,  is  presented  in  Yilmaz's  Ph.D.  thesis. (32) 

During  the  first  half  of  1978,  we  tried  to  add  X25  software 
and  hardware  to  our  Intel  8080  facsimile  terminal.  In  the 
end  both  activities  only  had  useful  training  value.  Although 
the  exercise  was  successful,  the  resultant  software  was 
so  large,  that  it  was  impractical  to  have  it  coexist  with 
our  facsimile  software.  Our  hardware  was  purchased  as  a 
Microcomputer  in  1975,  and  could  not  support  multiprocessors. 
Others  (e.g.  the  European  Informatics  Network  Executive) 
have  also  found  that  a  multiprocessor  architecture  is  more 
appropriate  with  this  generation  of  microprocessor  for 
X25  based  applications. 

A  Digital  Facsimile  device,  a  Dacom  450,  was  obtained  during 
the  year.  This  device  has  both  a  built-in  modem  and  a  V24 
modem  interface.  The  block  structure  used  in  that  machine 
(33)  is  such  that  it  can  be  used  by  running  the  HDLC 
interface  built  for  our  LSI-lls  in  a  transparent  way.  The 
interface  has  to  be  run  in  a  strange  block  synchronous  mode; 
it  has  to  synchronise  in  a  6-bit  mode  and  then  switch  dyna¬ 
mically  to  8-bit  in  a  data  phase.  However,  it  has  proved 
quite  feasible  to  control  the  device  from  a  local  LSI-11. 

By  interrupting  the  Clock  signal,  it  is  even  possible  to 
halt  the  machines  for  flow  control  purposes  -  which  was 
not  feasible  on  the  earlier  6K  machines. 

We  have  defined  a  simple  set  of  procedures  for  storing  and 
retrieving  facsimile  files.  These  procedures  fit  above 
a  standard  transport  level  of  protocol.  The  system  is 
illustrated  in  Fig.  6.1. 


Fig.  6.1  Schematic  of  Terminal  Facsimile  Service 
IFT  =  Intelligent  Facsimile  Terminal. 


22. 


We  are  building  the  software  in  a  modular  form  so  that  the 
network  interfaces  to  IFT  can  be  either  a  standard  ARPANET 
1822  or  a  standard  HDLC  one.  The  network  access  and  transport 
software  will  be  the  ARPA  Internet  TCP  for  the  first  case, 
and  X25  plus  transport  level  for  the  second.  Thus  we  hope  to 
develop  a  genuinely  versatile  terminal.  We  have  started 
developing  standard  software  for  filing  and  retrieving  single 
papers  for  three  computers  above  the  TCP  level;  these  are 
the  DEC  TOPS  20,  the  PDP-11  with  UNIX,  and  the  PDP-11 

with  the  MOS  operating  systems. 

Based  on  these  building  blocks,  we  hope  to  be  able  to  provide 
a  prototype  service  over  ARPANET  /Satnet  by  the  summer  of  1979, 
and  over  Euronet  by  late  1979. 

A  new  facsimile-related  project  is  starting  in  1979,  with  the 
Sim  of  integrating  facsimile  with  other  office  automation 
functions . 


VII.  SIMULATION  ACTIVITIES 


7.1  Introduction 


Earlier  annual  reports  have  mentioned  the  simulation  activi¬ 
ties  of  the  group.  However,  they  were  not  in  a  form  to  be 
described  easily  in  our  annual  report.  Now  two  publications 
have  been  produced  on  this  work.  These  are  reproduced  as 
Sections  7.2  and  7.3.  In  the  first  we  compare  the  relative 
advantages  of  Hop-by— Hop  and  end-to-end  procedures  —  using 
extensive  simulation  techniques.  In  the  second  more  detailed 
computations  were  used  to  contrast  different  end-to-end 
transmission  strategies  for  the  TCP  protocol. 


h 


7.2  Comparison  of  the  Hop-by-Hop  and  Endpoint 
Approaches  to  Network  Interconnection 
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This  paper  conpares  two  principle  architectural  approaches 
to  network  interconnection.  These  are  the  Endpoint  approach, 
which  uses  a  standard  internet  end-to-end  protocol,  and  where 
network  support  can  be  minimal;  and  the  Hop>-by-Hop  approach, 
in  which  internet  calls  consist  of  a  sequence  of  local 
virtual  calls  across  each  intermediate  net.  These  approaches 
are  first  contrasted  very  generally  and  certain  situations 
are  identified  which  heavily  favour  just  one  of  them. 
Concentration  specifically  on  flew  control  follows.  A  major 
point  to  emerge  is  that  an  Endpoint  architecture  is  likely 
to  demand  more  resources  fran  Hosts;  whereas  a  Hop-by-Hop 
Architecture  danands  more  from  Gateways  and  may  produce 
inferior  service.  Such  differences  are  illustrated,  for  a 
particular  internet  connection,  by  simulation  and  analysis. 


1.  INrHGDUCriC»4 

Interconnection  of  packet  switching  networks  has  occured  only  recently,  and  to 
a  limited  extent.  It  is  intended  to  provide  a  wider  selection  of  user  services, 
by  creating  in  effect  an  enlarged  net  -  which  will  be  referred  to  here  as  a 
"Catenet"  (Pouzin  74).  In  this,  individual  nets  are  joined  at  their 
boundaries  by  what  have  cane  to  be  known  as  "Gateways"  (Cerf  74,  Pouzin  74, 
Sunshine  77a).  One  exanple  of  an  operational  Catenet  is  the  "TTII*  Catenet" 

(Cerf  78c,  Bennett  79)  linking  the  ARPANCT  (Roberts  73)  to  a  nunber  of  private 
datagram  nets,  including  SATTIET  (Jacobs  78),  the  ARPA  Packet  Radio  Net 
(Kunzelman  78),  and  the  Xerox  PAK  Ethernets  (Metcalfe  76).  Another  exanple 
is  the  EIN  Catenet  linking  the  EIN,  CYCLADES  and  NPL  nets  (Departs  76). 

A  Gateway  can  be  regarded  -  rather  gt'nerally  -  as  an  entity  -  or  entities  - 
interfacing  connected  nets,  and  residence  within  such  nets  is  not  precluded 
(Sunshine  77a).  There  is  general  agreement  that  Gateways,  which  rtiside  outside 
the  nets  they  interconnect,  should  interface  to  these  nets  as  Hosts,  rather 
than  say  as  Packet  Switches  (Lloyd  75,  Sunshine  75,  Sunshine  77a).  'Diis  has 
lead  to  a  standard  model  of  a  Gateway  as  a  single  physical  entity,  attached  to 
two  or  more  networks  as  a  Host  (R>uzin  74,  Binder  75,  Gien  75,  Lloyd  75, 

Walden  75).  The  logical  Host  portions  in  such  a  Gateway  are  sometimes 
referred  to  as  "Gateway  Halves"  (Sunshine  77a),  which  we  shall  abbreviate  to 
"Gate  Halves".  Figure  1  illustrates  such  network  interconnection,  and  we  shall 
assime  this  model  in  the  rest  of  this  paper. 


I 


Figure  1.  Interconnection  of  Networks  through  a  Gateway 


Opinion  is  divided  on  the  protocol  level  at  which  traffic,  on  individual 
internet  end-to-end  connecticms  should  be  transferred  throu^ 

Gateways,  between  the  Gate  Half  portions.  The  two  main  alternatives  are 
transfer  at  the  datagram  level  using  a  standard  internet  datagram  format 
(Pouzin  74,  Gien  75),  or  transfer  at  the  Virtual  Call  Level  (Gien  75,  Lloyd  75, 
^nshine  77a).  Transfer  ai  an  even  hi^er  level  -  e.g.  the  Virtual  Terminal 
Level  -  is  also  possible  (Higginson  75),  These  alternatives  lead  to  two 
distinct  architectural  approaches  to  network  interconnection,  which  have  been 
labelled  "Endpoint"  and“Hop-by-Hop"  (Sunshine  77a). 

In  the  Endpoint  approach  -  illustrated  in  Figure  2  -  internet  traffic  transfer 
through  Gateways  is  at  the  datagram  level,  and  Hosts  must  use  a  standard 
internet  end-to-end  protocol  -  such  as  TCP  (Cerf  78a)  or  INWG  96  (Cerf  78b). 
Transfer  of  internet  traffic  between  adjacent  Gateways  and  Hosts,  may  then 
use  minimal  local  net  services.  In  the  Hop-by-Hop  approach,  illustrated  in 
Figure  3,  no  standard  end-to-end  protocol  is  required;  internet  traffic 
transfer  throu^  Gateways  is  at  the  Virtual  Call  level.  Here  internet  connections 
are  conposed  of  a  chain  of  local  virtual  calls  across  intermediate  networks, 
and  Gateways  may  need  to  translate  between  different  virtual  call  protocols. 

This  approach  also  permits  Application  Level  protocols  (e.g.  Virtual  Terminal 
Protocol)  to  be  translated  at  Gateways  (from  one  net  to  another)  if  there  is 
no  internet  standard. 

This  paper  is  concerned  with  a  oonparison  of  these  approaches,  particularly 
with  regard  to  flow  control.  We  begin  (Sections  2  and  3)  by  corparing  the  two 
approaches  very  generally,  and  show  that  some  situations  heavily  favour  just 
one  of  them.  Concentration  specifically  on  flow  control  follows  (Section  4) 
and  is  deemed  most  relevant  to  situations  in  which  either  ^proach  is  feasible. 

A  major  point  to  emerge  is  that  an  Endpoint  architectxire  is  likely  to  demand 
greater  resources  from  Hosts,  whereas  a  Hop-by-Hop  Architecture  denands  more 
frcm  Gateways  and  may  produce  inferior  service.  Such  differences  are  illustr¬ 
ated  for  a  particular  internet  connection  by  simulation  and  analysis 
(Sections  5  and  6). 


2.  ENDPOINT  INTEHOONNECTia^ 

Endpoint  interconnection  is  used  in  both  the  TCP  Catenet  -  with  TCP  as  ♦^he 
internet  end-to-end  protocol  -  and  in  the  EIN  Catenet,  which  uses  the  EIN  end- 
to-end  protocol  (Departs  76).  Its  main  advantage  is  in  enabling  sinple 
Gateways.  These  are  not  required  to  handle  individual  virtual  calls,  and  may 
use  local  net  datagram  services  -  where  possible  -  to  transfer  internet  packets. 
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Figure  2.  Tlie  Endpoint  Interconnection  Approach 


Figure  3»  The  Hop-by-Hop  Interconnection  Approach 


Another  advantage  is  that  inter-Gateway  routing  can  be  dynamically  altered,  on 
a  packet-by-packet  basis,  allowing  rapid  response  to  network  failures  and 
congestion. 

The  main  disadvantage  is  the  requiranent  of  end-to-end  standardisation.  Local 
nets  are  likely  to  have  iirplemented  their  own  Host  level  protocols.  The 
addition  of  an  internet  standard  will  not  cmly  be  costly  to  inplement,  it  may 
also  leave  some  "disadvantaged"  Hosts,  unable  to  engage  in  internet  cornnun- 
ication.  This  has  occured  in  both  the  TCP  and  BIN  Catenets  (Kirstein  78, 
Deparis  76),  and  has  required  the  construction  of  special  Hosts  -  known  as 
"internet  service  sites"  (Sunshine  77a)  -  which  inplement  the  internet  protocol 
on  behalf  of  those  disadvantaged  Hosts,  in  their  own  nets.  Other  disadvantages 
of  this  approach  include  the  likelihood  of  large  packet  headers,  which  would 
cut  into  usable  comnunication  bandwidth  (Postel  78),  and  the  unnecessary 
dvplication  by  Gateways  of  end-to-end  features,  when  internet  traffic  passes 
over  a  virtual  circuit  net. 

The  Endpoint  approach  is  favoured  mainly  for  interconnecting  datagram  networks 
-  such  as  in  the  BIN  and  TCP  Catenets  -  where  there  is  agreement  on  end-to-end 
standardisation . 


3.  HOP-BY-HOP  INTEROGNNBCTICW 

There  are  two  distinct  applications  of  the  Hop-by-Hop  ^proach.  The  first. 
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wrtiich  we  mention  only  briefly,  involves  interconnection  of  public  packet 
switching  networks,  according  to  the  recently  proposed  interconnection  standard 
X.75  (OCITT  78).  Here  virtual  calls  originating  in  one  network  are  relayed  - 
by  means  of  a  network  processor  known  as  an  SH:-  directly  to  an  equivalent  SIE 
in  an  adjoining  network;  and  one  might  regard  such  SIEs  as  analagous  to  Gate 
Halves  in  the  more  conventional  Gateway  model  (Section  1).  Clearly,  however, 
there  is  no  question  of  using  the  Endpoint  approach  here. 

Another  application  of  the  Hop-by-Hcp  approach  is  in  interconnecting  networks, 
which  (in  general)  possess  different  virtual  call  services.  A  major  function 
of  Gateways  may  then  be  in  translating  between  such  services.  Ibis  approach 
has  been  used  to  interconnect  two  X.25  networks  -  Datapac  and  Telenet 
(Rybczynski  78) ,  and  to  interconnect  ARPANET  and  EPSS  (Higginson  78) .  We 
proceed  to  discuss  this  application  in  more  detail. 

The  main  advantage  of  the  above  Hop-by-Hop  application  is  to  allow  the  use  of 
existing  virtual  call  services  for  making  internet  connections.  Another 
advantage  is  that  unlike  the  Enc^int  approach,  long  internet  end-to-end 
packet  headers  are  not  required,  and  packet  headers  could  be  made  very  short 
using  abbreviated  addressing,  with  full  addresses  kept  in  Gateway  tables 
(Lloyd  75).  The  main  disadvantage  of  this  application  is  the  corplexity  of 
performing  protocol  translation  and  the  possibility  of  not  being  able  to  map 
all  the  functions  of  one  virtual  call  service  into  the  functions  of  another 
(Gien  75). 

The  Endpoint  approach  may  also  be  used  for  this  application,  provided  an 
agreement  on  end-to-end  standardisation  can  be  reached;  this  approach  is 
preferred  by  some  authors  (Pouzin  74,  Gien  75,  Walden  75).  However,  as  we 
have  seen,  both  approaches  possess  advantages  aivi  disadvantages,  and  in  a 
practical  situation  one  would  have  to  assess  the  relative  inportance  of  these 
as  well  as  of  other  relevant  criteria.  In  the  rest  of  this  paper,  we  concen¬ 
trate  on  one  such  criterion,  flow  control,  which  has  previously  beep 
neglected . 


4.  FLOW  CONIHDL  CJIARACIERISTICS  OF  NEIWC«K  INIERCDNNBCTICIN 

The  objectives  of  flow  control  have  been  defined  as  the  provision  of 
adequate  end-to-end  service  (throughput  and  delay)  and  the  minimisation 
of  the  relevant  network  resources  (storage,  transmission  bandwidth, 
and  processing  overhead)  (Pouzin  76).  With  network  interconnection, 
one  interconnection  sqproach  is  stperior  to  another  if  it  provides  better 
service  using  the  same  resources  -  or  equivalently  needs  fewer  resources  for 
the  same  service  level.  We  proceed  to  evaluate  the  Enc^int  and  Hcp-by-Hop 
(including  X.75  Hop-by-Hop)  approaches  in  this  respect.  Criteria  of  interest 
will  conprise  end-to-end  delay,  throughput  limitation,  and  Host  and  Gateway 
resource  requirements.  Consideration  is  also  given  to  the  requirements  of 
certain  traffic  types  (e.g.  bulk  traffic). 

We  look  first  at  the  Endpoint  approach.  When  used  to  interconnect  datagram 
networks,  service  could  be  optimal,  both  because  packets  can  be  delayed 
minimally  at  Gateways  and  because  the  ability  to  use  adaptive  routing  allows 
flexible  response  to  internet  and  local  net  congestion.  However,  service  would 
be  sub-optimal  whenever  traffic  had  to  cross  a  (e.g.  public)  virtual  circuit 
net,  due  to  unnecessary  local  net  sequencing  of  internet  traffic  (Sunshine  77a). 

Resource  requirement  in  the  Endpoint  approach  inside  Hosts,  both  in 

terms  of  storage  use  and  processing  ti^,  is  likely  to  be  larger  than 

in  the  Hop-by-Hpp  iqproach.  This  is  because  end-to-end  internet  I 

protocols,  such  as  liP  and  INWG  96,  assune  minimal  subnet  level  service,  ' 
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H;id  SI)  rtxiuiiX"  vn*  jind  ccnijlox  Ikwl  attlion.  f'urihi'r,  such  i)i\)t<v<>ls 

nxiuire  internal  Host  stora^^o,  both  for  packet  rctruiMiiission  (at  a  scrKh?r) 
and  for  ni*ss!i4;i*  rc:issnit)ly ,  st.*quenciug  and  delivoi'y  (at  a  n^ceivor).  Sueb 
stora*;t*  is  relattxi  to  the  juiuunt  of  outstanding  (unackixw lodged)  traffic  at  a 
st'nder,  and  w)uld  lx*  jMirt  icularly  largi^  for  bulk  (or  real  tini‘)  data  tninsfers 
over  U)i^  internet  coiux‘ctions.  In  contrast,  tlie  functit>ns  of  Gateways  can 
be  loss  coiiplex,  and  buffer  storage  for  pacUeet  retriuiMnission  mid  stxiuencing 
(if  ustHl  at  all  by  Gatexvays)  would  be  requirtxl  only  over  single  nets,  wfien* 
the  amount  of  outstanding:  traffic  would  be  liniittxi. 

'ITie  main  feature  of  the  Hop-by-Hop  appnxich,  fixm  tlie  point  of  view  of  Hosts, 
is  to  make  internet  coiunxttions  look  like  local  net  connections  witii  re'spex't 
to  rt‘source  utilisation.  'Hius,  Host  storage*  is  only  rexiuire*d  to  suppeirt 
traffic  ove*r  a  single*  ne*t,  and  would  be  leiss  -  particularly  for  bulk  data 
transft*r  -  than  in  the  Enc^xiint  approach.  Further,  local  prottxols  exiuld  lx* 
suipler  and  less  time-consuming  to  mataige  than  a  cxjtTple*x  internet  protocol, 
and  it  would  Ixj  unnex*t's.sary  to  miuiagt*  simultsuiexiusly  two  (Iccnlft  inti*rm*t) 
protocols . 

Conversely,  Gatt'ways  in  the  Hop-by-Hop  approach  meust  maintain  Individual 
virtual  calls  for  i*at:h  intt*rnet  conntx'.tion  thc'y  handle.  Tliis  inplie*s  gre'ate*r 
prccessing  mid  storage*  rexjui  reorients  than  in  thc^  Endpoint  ajiproach.  Fur  the 'more*, 
stu'vicx'  in  the  Hop-by-Ho[i  approach  is  liktely  to  be  .sub-optimal,  due*  Ixith  to 
unnecessary  se*quencinj:  on  each  virtual  call  hop,  and  to  c'Xtra  dc'lays 
asscK'iated  with  connt*c‘tion  rruuiagement  and  protoex)!  tnui.slation.  A 
nc*cc.sslty  for  proUxxil  trmislation  miglit  also  limit  the*  Ihioughput  olita inalile* 
ac:ross  a  C>ate'way. 

These  observat  lexis  inply  seve?ral  tradexiffs  bt*tween  the  Endpoint  and  Hop-by-Ikip 
approaches.  The  Endpoint  approach  is  c^apable  of  be*tter  st*rvice,  re*quirc*s 
ft?wor  Gateway  resources,  but  requires  more  Hexst  rccsources;  whc*ix*as  the*  Ikip-by- 
Hop  approach  limits  Host  rt?sourcc  requirements  to  those  for  oixiinary  Icxal  net 
cxinnect ions .  We*  demonstrate  those  tradeoffs  in  the  following  sex'tiexis,  using 
a  simulation  mndol  of  an  internet  connection. 


5.  IWIFRNLT  SIMIUJITICW  MXEL 

In  this  section  we  describe  a  simulation  mode*l  of  an  interne*!  e*nd-to-e*nd 
exinnexrtlon  between  two  eximiunlcating  pro(x*sses.  Thc^sc*  are  resident  in  Hosts, 
se»parated  by  "n"  intermediate  ne.*ts,  and  "n-1"  Gate*ways  -  Figure*  4  -  and  a 
fixed  internet  route  is  assimcKi.  Each  Gateway  is  cenposed  of  two  Gate*  Halve's, 
according  to  the  model  of  Sectlcxi  1.  Tlwo  levels  of  protexxil  are  included; 
an  internet  end-to-end  level  between  the  two  Hosts,  and  a  l(x?al  net  virtual 
cmll  level  between  adjaceint  Gate  Halves,  which  we*  shall  refer  to  as  the  Gateway 
level.  The  latter  protoexil  also  ope*rates  between  e*ach  Host  and  its  lex'al 
Gatejway,  and  so  we  consider  each  Host  to  contain  a  logical  Gate  Half  portion, 
as  well  as  a  logical  end-to-end  portion.  Different  mLX'hani^ms  will  be*  ascxl  at 
each  of  these  protocol  levels  to  enable  modelling  of  both  the*  Endix^int  mid 
Hop-by-Hop  approaches. 


5.1  PHOTOOM..  MECHANISMS 

The  protcxol  mechanisms,  which  can  be  use*d  at  each  level,  cxirprise  the 
following: 


liia-: 


Kiid- to- End  I’lotorol 


Sourc* 

Process 


f 

# 


[1 


i  lint  ion 
Prorc'RB 


Host 

End- to- End 
Port i on 


O  :  Cnto  Hnlf 


Figure  4.  The  Internet  Siimilntion  Model 


Nil  :  No  I>rotoool 

PAR  ;  I\>sltivt'  Ackncwhxlgtnii^nt  and  Retranunission 

SPAR  :  StHjuc'ncing,  R>sltlvp  Acknc)wlc>dgntK>nt  and  Ri‘tran«nlssion 

We  shall  denote  by  the  term  "Nil"  the  ixmoval  of  either  level  of  protocol. 

The  "PAR"  protocol  (Sunshine  75)  operates  bt>twoen  a  Sender  and  a  Riveiver  of 
packets.  Each  packet  traniflnitt^  by  the  Sendt^r  is  alUxyited  a  unique 
identifier,  and  a  copy  is  retained  for  retraiv^nission  using  internal  buffer 
storage.  The  Receiver  imni'diately  acknowledges  each  correctly  rec.eiv(Hl  jxicket, 
by  returning  its  identifier  in  a  special  ackrKwhxlgement  packet.  AcknowlixigiHl 
packet  copies  are  discarded  at  the  Sender,  arxl  unac.knowledgtHi  packets  an.' 
retran.'anitted  (repeatedly  if  necessary)  after  a  tlmt'out  interval. 

The  "S^AR"  protocol  (Sunshine  75)  is  a  more  conplex  version  of  the  "PAR" 
protocol.  Trananltt^  packets  are  allocated  a  monotonic  incn'asing  sequt'nce 
nurber,  and  packets  arriving  at  the  Receiver  are  re-orden>d  in  sequence,  using 
internal  storage.  Whenever  possible,  single  packets  or  contiguous  packt't 
blocks  are  delivered  in  sequence  by  the  Receiver  to  the  next  stage,  and  a 
single  acknowledgement  packet  is  returned,  carrying  the  sequence  of  the  ne^t 
packet  to  be  delivered.  This  acknowledges  all  preceding  packt't  delivi'ries. 

The  Receiver  also  recognises  and  discards  duplicate  packet  arrivals  (for 
packets  already  delivered  or  being  re-ordered)  and  a  positive  ackncwledgmu'nt 
is  returned  to  the  Sender,  re-acknowledging  previous  deliveries.  The  retrans¬ 
mission  procedure  at  the  Sender  is  identical  to  that  in  the  PAR  protocol. 

The  PAR  protocol  is  intended  mainly  for  efficiency,  to  ix'-supply  lost  packets, 
and  it  can  introduce  packet  duplicates.  Such  a  pixjtoool  is  used,  for  exanple, 
betWE?en  packet  switches  in  the  ARPANET  (McQuillan  77).  The  SPAR  protocol  is 
the  basis  of  a  Host  level  virtual  call  protocol,  since  it  delivers  packets  in 
sequence,  without  loss  or  duplication  -  in  the  absence  of  system  crashes  and 
provided  certain  constraints  on  the  re-ust'  of  sequence  niirtx'rs  are  obt'ye'd 
(Sunshine  75).  We  shall  use  it  to  represient  both  an  Internet  end-to-end 
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prx)lcK'ol,  and  a  Giiti-way  level  loeal  virtird  call  prDltHX)! .  IVfausi-  e/ in>ar)si>ns 
t)f  diffennit  appiXKU"hc*s  are  niride  at  a  umslant  thr\)ut?hput  (i.e.  ix)iLst:uil 
st^rvice)  level,  jidditional  mr'cluunsjns  for  flow  constraint  -  e.g.  a  window 
(Cerf  74)  -  are  not  used  at  eitlier  pixjtocrol  level. 

Wi*  shall  denote  the  protocol  mix  ased  in  the  simulation  model  by  the 
convention  A(B1,B2,  ....  Bn),  with  "A"  dentJting  tlie  end-to-c*nd  mxrhani.*^  and 
Bi  denoting  the  Gateway  level  i«?chani.<3n  acn.)s.s  the  i’th  net. 


5.2  MUDKL  DRSCKIFFION 

Data  flow  in  the  model  is  one  way  from  a  "Source  Process"  and  "Source  Host"  to 
a  "Destination  Host"  and  "IX'stination  Process".  The  Gateway  level  piotcxol 
supports  both  forward  data  traivsfer  and  the  return  of  end-to-end  acknowlixlge- 
nonts.  We  assume  the  two  protocol  levels  act  independently,  with  no  piggy¬ 
backing  of  Gateway  level  acknowlodgunents  on  end-to-end  packets. 

We  describe  the  model  in  terms  of  its  three  constituent  levels:  end-to-end. 
Gateway- to-Gateway,  and  subnet  -  Figure  4.  We  assime  that  (at  Gateways  or 
Hosts)  packets  are  transferred  between  these  levels  instantaneously,  and  we 
are  not  concemtxi  with  blocking  or  processing  delays  by  any  level.  In  the 
following  ck'sc'Tiption,  the  end-to-end  and  Gateway  levels  are  assumed  inplicitly 
to  inplanent  one  of  the  protocols  described  in  Section  5.1. 

At  the  end-to-end  level,  the  input  is  a  steady  stream  of  messages  arriving  at 
the  Stxu'ce  Host  fixm  the  Source  Process,  with  constant  inter-message  arrival 
time  A.  Each  message  is  encapsulated  in  a  single  packet  and  passed  (inmediately) 
to  the  next  level  (the  Host  Gate  Half  portion).  At  the  Destination  Host,  the 
messages  in  packets  handed  over  by  the  Host  Gate  Half  portion,  are  passed  - 
possibly  after  sequencing  -  to  the  Destination  Process  (assim^  always  able  to 
accept  them),  and  a  positive  acknowledgonent  may  be  handed  back  to  the  Host 
Gate  Half  for  reverse  trananission. 

The  Gate  Halves  of  the  Gateway  level  take  their  input  from,  and  send  their 
output  to,  either  an  adjoining  Host  end-to-end  portion  (in  the  Hosts),  or  an 
adjoining  Gate  Half  (in  the  Gateways).  Gate  Halves  are  assuned  not  to  fra^nent 
packets.  Packet  transfer  between  opposite  Gate  Halves  is  through  the  subnet 
level.  This  inposes  a  variable  delay  and  the  possibility  of  loss  on  each 
packet  transferred.  An  Erlang  distribution  is  used  to  m^el  the  delay  of  each 
subnet.  This  has  a  probability  density  function  f,  which,  for  a  delay  x,  is 
given  by: 

f(x)  =  (k.u)*^.x^~^.e"*^/(k-l)!  x>=0  (1) 

The  jjarameter  1/u  is  the  mean  delay,  and  k“i  is  the  delay  coefficient  of 
variation.  When  k  is  1,  f(x)  is  the  exponential  distribution.  Larger  k 
produces  a  distribution  with  a  sharp  peak  near  the  mean  and  with  a  long  tail 
extending  to  x  equals  infinity.  This  correspemds  to  the  types  of  delay 
encountered  in  practice  (Kleinrock  77,  Gien  78).  We  shall  set  k  to  25  in  all 
cases,  representing  lew  delay  variation  (the  corresponding  delay  coefficient 
of  variation  is  .2).  Loss  of  packets  in  transit,  or  loss  due  to  insufficient 
buffer  space  at  Gateways,  is  represented  by  applying  a  fixed  probability  of 
loss  p  independently  to  each  packet  in  transit.  The  delay  and  loss  in 
each  subnet  are  identical  for  packets  and  ackncwledgnents  travelling  in 
either  direction.  We  note  that  for  high  packet  loss,  the  exact  value  of  k, 
used  for  eq'l,  will  be  less  significant,  since  retrananission  (following 
packet  loss)  will  then  be  a  major  cause  of  long  packet  delay. 

The  following  suimorises  the  parameters  used  in  the  model: 


A  :  Inter-arrival  tine  of  messages  at  the  Source  Host 

R  :  Source  Host  end-to-end  retrananission  timeout 

Ri  :  Gateway  level  retransmission  timeout  across  the  i'th  net 

ki  :  x-alue  of  k  in  eq'l  for  the  i'th  net 

Ui  :  value  of  u  in  eq'l  for  the  i'th  net 

Pi  :  packet  loss  probability  in  the  i'th  net 

Performance  measures  for  the  model  ocnprise  the  end-to-end  message  delay, 
the  retrananission  overhead  at  each  protocol  level,  and  Gateway  and  Host 
storage  requirements.  Appendices  1  and  2  present  an  analytical  derivation  of 
certain  of  these,  under  various  protocol  mixes,  and  this  allows  partial  valid¬ 
ation  of  simulation  i-esults. 


6.  SIMULATION  RESULTS 


Ihe  conclusions  of  Section  4  are  illustrated  with  sinulation  results,  using  a 
GPSS  program,  for  the  internet  connection  model  of  Section  5.  Where  possible, 
analytical  results  (Appendices  1  and  2)  are  used  to  validate  corresponding 
simulation  results.  H/hen  these  agree  closely,  only  the  (more  accurate) 
analytical  results  are  shown  in  the  graphs  below.  Otherwise,  both  sets  of 
results  are  shown.  Analytical  results  are  identified  with  an  asterisk. 


We  look  at  an  internet  connection  spanning  two  identical  nets,  and  carrying 
bulk  traffic,  for  which  resource  requirements  are  significant  (Section  4). 

The  following  parameter  settings  are  used,  with  the  average  end-to-end  subnet 
delay  (l/uj  +  1/U2  )  set  to  one  unit  of  time: 


A 

R 


1*1, k2 
1*1,% 
l/Ui,I/U2 


.2 

3  without  Gateway  retraiv^ssion 

oo  ;  with  Gateway  retransmission 

25 

1.5 

.5 


The  message  inter-arrival  time  A  corresponds  to  the  production  (and  trananission) 
of  5  messages  in  an  average  end-to-end  subnet  delay.  Ihis  represents  bulk 
traffic  on  a  connection  where  the  packet  trananission  delay,  into  (and  out  of) 
the  Catenet,  occxpies  a  anall  fraction  of  overall  end-to-end  delay  -  e.g.  where 
several  padtet  switches  or  a  satellite  link  separates  the  intermediate  Gateway 
from  either  Host.  The  retrananission  timeouts  -  R,R<,R2  -  are  set  to  3  tiroes 
the  average  transit  mediun  delay  of  the  level  over  which  they  operate.  This 
value  has  been  found  to  minimise  retrananission  delay,  subject  to  minimising 
the  nunber  of  retrananission  diplicates  produced,  in  this  type  of  simulated 
connection  (Edge  78).  A  wide  range  of  packet  loss  rates  is  used,  representing 
connections  varying  from  the  very  reliable  to  heavily  congested  ones  with 
freqiient  packet  discard  from  Gate  Halves  or  in  each  subnet. 


The  End^int  and  Hoi>-by-Hop  approaches  are  represented  by  the  following 
protocol  mixes  (Section  5.1): 

Hop-by-Hop  =  Nil(  SPAR,  SPAR) 
Endpoint  with  Gateway  retrananission  =  S>AR,(PAR,PAR) 
End^int  without  Gateway  retrananission  =  SPAR, (Nil, Nil) 


The  use  of  Gateway  level  retransmission  in  the  Endpoint  approach  has  been 
propxjsed  to  improve  efficiency  over  lossy  nets  (Binder  75,  Walden  75),  althou^i 
it  does  require  additional  Gateway  storage  to  keep  packet  copies  for  retrans¬ 
mission.  Analysis  (Metcalfe  73)  has  shown,  for  exanple,  that  such  intermediate 
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^Vlrlu\^^uisKU)n  ixh1vk.’i's  finl-lo-i'iid  ik-lay  wtnni  jxu'Jtel  \ik-;s  is  f  u  .uit  .  A 

of  the  ri'sults  prx'st'nttxl  hoix*  dtmnistmte  this  juid  other  drawtunks  of  iK>t 
using  GHteway  level  retnuv^nssion.  QtuMirisini  with  tht‘  Ho^v-by-Hop  jippirmeli 
s-ulis«.\iut>nt ly  assuifs  tlie  ast'  of  G^lttx^•ay  n‘tnul^^llission  with  tlie  Ktid|K>ii)t 
H|.ipi\xu'h,  witienever  paeket  loss  is  significant. 

Figure  5  slxiws  the  vai'iation  of  the  avx^rage  end-to-ixid  iik'ssage  delay  (fixm 
ni‘ssagi‘  appi'arance  at  the  SLiurce  H^'st  to  delivery  to  tht*  Destination  Pixvoss) 
against  the  packet  loss  (pj  luid  P2)  in  each  subnet.  We  note  that  analytical 
validation  for  the  Endpi>int  aj^iroach  witiiout  Gateway  Ri'liaiwiiission  i.s  i*xai't ; 
wheix'as  appmximat  ions  made  in  tht'  luiulysis  of  the  otlier  apjirtxicht's  (.stx* 
Appt'ndLx  2)  lead  to  ^jihII  dl f ferx'nix's  betwet'ii  txirresixMiding  analytical  luid 
simulation  rt'sults.  Figure  5  dcrxMvstrates  the'  higlier  delay  in  tin'  Kiui^xiint 
a|ipix>ach  wtien  Gateway  ix't  nuiymis-sion  is  not  ustxl,  that  wx'  nx'nt  ioiu'd  jiIkax'. 
niis  occurs  Ixx'aust'  end-to-<'nd  rx'tnuii^nission  has  to  ust'  a  hunger  ix'tnuis- 
mission  timeout  thiui  Gati'way  lewl  lx*tr^ul^^nission.  Q.«n)arison  of  the  Kndixnnt 
apprtiach,  using  Gnttxvay  rx'trajv^ission,  with  the  Ht'p-by-Hi>p  appixiach  shows  the 
latter  prodix'os  slightly  lai-ger  dt'lay,  wtu'n  jiacket  loss  tx-curs.  Altluxigli  Ixith 
of  thi'se  rx't r:uii^".i t  at  the  Gatt'way  level  with  idt'iitical  tirrxxiuts,  the  Hop-by- 
Hop  abroach  .stxiuences  packets  at  tlie  intermixiiate  Gate'way  (Ix'twxx'ti  either  net) 
in  addition  to  Dt'stination  stxiiK'ncing.  As  nx'ntiontxl  in  Sex'tiexi  4,  tJiis  ituxxses 
aelditional  imiKxx'ssary  ck'lay. 


Figure*  5.  Average  End- to- End  l>lay  v»r*ua  Tackrt  Lows 


(log  scale) 


Figure  6.  Average  Retransmissions  of  each  Packet  versus 
Packet  Loss 


Figure  6  shows  the  average  nintjer  of  times  each  packet  is  retransmitted  - 
either  at  the  end-to-end  level,  or  at  the  Gateway  level  -  in  either  net. 
Retrananission  by  the  Hop-by-Hop  approach  in  the  2'nd  net  -  not  slicwn  -  is 
marginally  greater  than  in  the  I'st  net.  Figure  6  shows  that  retransmission 
is  greatest  in  the  Enc^Joint  approach  without  Gateway  retrananission,  followed 
by  retransmission  with  the  Hop-by-Hop  approach.  In  the  former,  end-to-end 
acknowledgement  is  in  sequence,  and  so  tne  loss  of  a  single  data  packet  causes 
the  retransmission  (with  bulk  traffic)  of  a  large  nvxrber  of  outstanding 
.successor  packets,  whose  acknowledgements  have  been  held  up  (McKenzie  74, 

Edge  78).  Such  unnecessary  retransmission  also  occurs  in  the  Hop-by-Hop 
approach,  but  at  a  reduced  level,  since  at  the  Gateway  level  there  are  fewer 
outstanding  packets  to  be  held  up  and  unnecessarily  retransmitted.  The  lew 
level  of  retransmission  in  the  Ehdpoint  {q}proach  with  Gateway  retransmission 
occurs  because  packets  are  ackncwledged  out  of  sequence  -  by  the  Gateway  PAR 
protocol  -  ind^jendently  of  the  loss  to  other  packets. 

The  dependence  of  retrananission  iqxn  the  message  iiqsut  rate  (1/A  »  nurber  of 
messages  introduced  in  an  average  end-to-end  sutmet  delay),  at  constant  packet 
loss  (PiT2*'^)>  shown  in  Figure  7.  This  shows  that  an  advantage  of  the 
Dic^int  Approa^  (with  Gateway  retrananissiem)  is  that  retrananission  is 
invariant  to  message  ii^t  (i.e.  invariant  to  the  traffic  rate).  This  occurs 
because  packets  are  ackiKwledged  and  retrananitted  independently  of  others  by 
the  (kiteway  level  PAR  protocol  -  in  agreement  with  the  analysis  of  Appendix  1. 
conversely,  x^trananission  in  the  Hop-by-Hop  iqjproach  increases  as  the  message 
input  rate  rises  (i.e.  as  traffic  be<xmes  heavier)  because  on  each  packet  loss, 
there  are  more  outstanding  packets  available  to  be  needlessly  retrananitted 
(by  the  mechanian  described  earlier).  It  is  only  at  verj’  lew  mi's.sage  input 
(1/A<1)  -  characteristic  of  interactive  traffic  -  that  lost  packets  can  be 
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Figure  7«  Average  Retranamiaaions  of  each  Packet 
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retransmitted  and  acknowledged  before  later  packets  timeout.  In  this  case, 
the  retrananission  rate  approaches  that  of  the  Endpoint  approach. 

We  note  that  in  the  Endpoint  approach,  Gateway  retransmission  duplicates  fixmi 
the  first  net  can  enter  the  second  net,  thereby  increasing  the  average  niiiix>r 
of  message  duplicates  therein.  Hie  average  nui4x?r  of  n.'ssage  duplicates  per 
message  in  the  second  net  is  shown  in  Figure  7.  Use  of  a  sinple  duplicate 
filtering  scheme  at  Gateways,  as  used  between  nodes  in  the  AI^A  Packet  Radio 
Net  (Kunzelnan  78),  would  correct  such  "anplification"  of  duplicates. 

In  Figure  8,  the  total  average  storage  requirement  (for  data  packets)  for  both 
Hosts  is  shown,  in  terms  of  nuntiers  of  packets.  This  ocnprises  retrananission 
queue  storage  at  the  Source  Host  and  packet  re-ordering  storage  at  the 
Destination  Host,  for  both  the  Gateway  and  end-to-end  protocol  levels.  However, 
the  Source  Host  is  assumed  to  keep  only  one  packet  copy  for  retransmission  at 
either  level,  when  both  of  these  levels  operate.  Figure  8  shews  that  Host 
storage  requirement  with  the  Hop-by-Hop  approach  is  alntist  half  that  in  the 
Endpoint  approach,  for  low  packet  loss.  This  is  to  be  expected  since  Host 
storage  in  the  Hop-by-Hcq;)  approach  is  required  only  for  a  single  net  connection, 
and  one  would  expect  an  even  larger  difference  for  connections  spanning  more 
than  two  nets. 

We  note  that  the  occurence  of  excessive  Host  storage  requirement,  with  high 
packet  loss,  for  the  Endpoint  approach  without  Gateway  retransmission,  in 
Figure  8,  is  due  to  the  long  delay  of  end-to-end  retrananission  (noted  earlier), 
wtiich  allows  large  queues  of  unackncxvl edged  packets  to  accimilate  at  each  Hivst, 
following  packet  loss. 


Figure  8.  Average  Hoata  Storage  Requirement  veraua 
Packet  Loaa 


Ihe  average  storage  requirement  (for  data  packets)  in  the  intermediate  Gateway, 
ahich  separates  the  I'st  and  2'nd  nets,  is  shown  in  Figure  9.  This  oonprises 
packet  re-ordering  storage  for  the  Gateway  level  across  the  I'st  net  (Hop-hy- 
Hop  approach  only)  and  retransmissicm  storage  for  the  2'nd  net  (both  approaches) 
Storage  requirement  is  shcAvn  to  be  slightly  greater  in  the  Hop-by-Hcp  approach, 
when  packet  loss  is  significant.  It  would  also  be  greater  at  low  packet  loss, 
since  (kiteway  retransmission  -  and  thus  Gateway  storage  -  would  then  not  be 
i^equired  for  the  Endpoint  approach.  Such  hi^ier  storage  requirement  illustrates 
the  greater  Involvement  of  (jateways  in  the  Hop-by-Hpp  approach. 

Figure  10  confcines  the  two  previous  results,  and  shows  the  average  storage 
requirements  both  for  the  Hosts  and  for  the  intemnediate  Gateway.  The  value 
of  this  result  depends  upon  the  "costs"  of  Gateway  and  Host  storage  being  equal 
(if  they  are  not,  each  storage  conponent  nust  be  weighted  accordingly).  When 
costs  are  equal,  Figure  10  shows  that  the  Hop-by-Hpp  tq^oach  is  less  costly  - 
storage-wise  -  than  the  Endpoint  approach,  for  non-zero  packet  loss. 
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.gure  9*  Average  Intermediate  Gateway  Storage  Requirement 
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Figure  10.  Average  Hosts  and  Gateway  Storage  Requirement 


versus  Packet  Loss 
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7.  CUNaA'SlONS 

TTie  Endpoint  and  Hop-by-Hop  approacht‘s  tc  network  ijilercx)nne<.'tion  exfiibil  a 
nurber  of  significant  differences.  Many  of  these  are  political  or  u>chnical 
in  natu.’e,  and  in  sane  sitiuitions  heavily  favour  one  or  the  other  approjich. 
However,  when  the  networks  to  be  Interconnected  possess  different  virtual  call 
services,  and  with  no  prior  agrxxwnt  on  an  end-to-end  standard,  tht^n  either 
approach  is  feasible.  In  this  case,  the  {jerformuice  of  each  approach  with 
respect  to  flew  control  will  be  more  significant  to  the  overall  evaluation  of 
each.  The  Endpoint  approach  was  found  to  Ix^nc^fit  fron  asing  Gateway  level 
retransanission,  with  significant  packet  loss,  since  -  at  the  cost  of  extra 
Gateway  storage  -  this  reduced:  end-to-end  delay,  unntxossary  retran.^jnlssion, 
and  Host  storage  requirements.  The  main  characteristic  of  the  Endpoint 
approach  is  a  large  rc>quiranent  for  Host  resources  -  particularly  storage  for 
bulk  data  transfer.  The  Hop-by-Hop  aKJroac:li  was  si^ewn  to  demand  from  hosts  the 
same  resource  amount  as  would  be  required  for  a  local  net  connection.  However, 
it  required  greater  Gateway  resources,  provided  slightly  poorer  service,  and 
produced  more  retranssnission  when  paeJeet  loss  occured  than  the  Enc^wint  approach. 
We  e.xpect  that  the  significance  of  these  differenc*es  would  ciepend  on  the 
particular  serxicc*  nxiuirements  and  resourex's  available  when  particular  nets 
were  inter(x»nnected. 
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APPtMiIX  1  -  ANALYSIS  OF  SINQJE  NLT  PERFORMANCE 

Analysis  of  the  performance  of  a  single  net,  under  both  the  PAR  and  SPAR 
protoOTls  of  Section  5.1,  has  already  been  carried  out  (Sunshine  75, 

Suashine  77b).  Here,  we  briefly  outline  the  derivation  of  results  relevant  to 
the  simulation  model  of  Section  5. 

We  look  at  a  single  protocol  level  -  PAR  or  SPAR  -  operating  between  a  Sender 
and  Receiver  of  packets,  at  each  end  of  a  single  net  -  Figure  11.  An  arbitrary 
probability  de^ity  function  (pdf)  f  for  the  subnet  delay  is  assuned,  and 
its  ^rresponding  probability  distribution  function  (PDF)  will  be  denoted  by  F. 
A  suitable  unit  of  time  for  the  system  would  be  the  mean  of  this  delay. 

However,  such  normalisation  requires  modification  in  the  multi-net  case  (see 
Appendix  2).  We  successively  add  in  the  effects  to  packet  delay  (in^asured 
from  initial  transnission  at  the  Sender  to  delivery,  by  the  Receiver,  to  the 
next  stage)  of;  a  packet  loss  probability  q,  repeated  retrananission  (of 
unacknowledged  packets)  at  intervals  I,  and  packet  sequencing  at  the  Receiver 
(SP^  only).  We  assume  a  constant  interval  T  between  successive  packet 
arrivals  (and  hence  between  successive  packet  transmissions)  at  the  Sender. 

“nie  subnet  delay  pdf  can  be  modified  to  take  account  of  packet  loss  by 
introducing  an  impulse  function  for  infinite  packet  delay  -  i.e.  for 
packet  loss  -  (Sunshine  75).  We  denote  the  resulting  pdf  fq  and  PDF  PQ  for 
packet  delay  (x)  as  follows: 

fq(x)  =  lim  [(l-q).f(x).(l-H(x-c))  +  (1-q) .(l-F(c)) .d(x-c)  (2) 

+  q.d(x-c)]  x»=0 

Pa(x)  =  lim  [(l-q).F(x).(l-H(x-c))  +  H(x-c)]  x>=0  (3) 

c-»» 

with  d  =  Dirac  Delta  Function 

H  =  Heavyside  Iftiit  Step  Function 
H(y)  =  Cl  :  y>=0 
Co  :  y <  0 


Figure  11.  Single  Net  under  PAR  or  SPAR  protocol 
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TTie  pdf  g  and  PDF  G  for  ck.*lay  with  ri‘ir}ui>3nisslon  lu-r*: 

n 

G(x)  -  1  -  TRl  -  FXXx-k.I)) 

k*0 

n  n 

g(x)  -  H  fq(x-J.I).7r(l  -  FC}(x-k.I)) 

J=K)  k=0 

k?‘J 

with  n  ■  Ix/Ij 

The  PDF  H  for  delay,  with  stxiui'ncing  at  the  Rix^eiver,  is: 

H(x)  *  j7g(x  t-  J.T)  (6) 

J=^) 

The  above  analysis  also  applies  to  packet  round  trip  dt'lay  -  for  travi'l  to 
the  Receiver  and  back  to  the  St*nder  (e.g.  as  an  ackncwledgimnit  on  tlw‘  return 
trip)  -  provided  q,  f  ,F,  fq,FQ,g,G  in  ixi's  2  to  5  luv  replatxxl  by  their  mund 
trip  counterpai'ts,  wiiich  wx>  denote  by  adding  a  diusii  to  each. 

The  following  performancie  measui'es  ciui  bt?  derivt'd  frxm  the  alxjve  rt’sults: 


av.  PAR  1-way  ck'lay 

X 

(1  -  G(x)).dx 

'^0 

(7) 

av.  PAR  round  trip  delay 

X 

f(l  -  G'(x)).dx 

(8) 

av.  nunber  of  retransmis¬ 
sions  per  packet  with  PAR 

- 

Ed  -  G'(j.I) 

J-1 

(9) 

av.  nurber  of  packets  in 
PAR  retransmission  queue 

X 

av.  PAR  round  trip 
delay/T 

(10) 

av.  SPAR  1-way  delay 

X 

no 

f  (1  -  H(x)) .dx 
*^0 

(11) 

av.  nunber  of  packets  at 
SPAR  Receiver 

- 

(av.  1-way  SPAR  delay  (12) 

-  av.  1-way  PAR  delay) /T 

(4) 

(5) 


Equation  8  gives  the  average  packet  acknowledgement  delay  for  the  PAIt  protixx)! . 
Equations  10  and  12  use  Little's  rt>sult  (Little  61)  for  the  aviu'age  nuntx'r  of 
customers  in  a  queueing  system,  given  their  averagi*  waiting  time  and  arrival 
rate.  We  note  that  none  of  the  performance  measures  obtalmxl  for  tine  PAR 
protocol  requires  constant  inter-packet  arrival  tine  at  the  Sendc'r  (T  in  eq'lO 
can  be  the  average  inter-arrival  time);  and  the  PAR  retran!:«nission  overiiend 
(eq'9)  is  independent  of  T  (i.e.  independent  of  tlie  traffic  rate). 


APPliNDIX  2  -  ANA1.YSIS  OF  MULTI -NLT  TOfftKMANCE 

Here,  we  look  at  the  simulation  model  of  Section  5,  v/'ich  comprises  "n" 
concatenated  nets,  with  a  single  end-to-end  protocol  level  and  a  Gateway 
protocol  level  across  each  net.  When  the  Gateway  level  is  not  ased,  the 
end-to-end  level  effectively  operates  over  a  single  net  (whose  delay  is  the 
sun  of  the  delays  over  the  n  Individual  nets),  and  performance  measunvs  may 
be  obtained  from  Appendix  1.  Otherwise,  there  are  n  concatenated  PAR  or 
SPAR  stages,  with  an  end-to-c^nd  protocol  over  thun  -  Figure  12.  The  avenige 
inter-packet  arrival  interval  at  the  i'th  stage  is  denoted  by  Aj.  Ifere,  a 
suitable  unit  of  time  for  the  system  is  the  sun,  over  the  n  stages,  of  the 
mean  subnet  delay  of  each  stage  (in  the  absence  of  packet  loss).  Such 
normalisation  is  used,  for  exanple,  in  Section  6. 


Arrival 

Interval 


Kftd- to-Knd  Protocol 


Figure  12.  End-to-End  Protocol  over  n  PAR/SPAR  Stages 


We  loc*  first  at  the  protocol  mix  SPAR(PAR,PAR,  ...  ,  PAR)  -  used  to  represent 
the  En^Joint  approach  with  Gateway  retransmission  in  Section  6.  We  assunc? 
there  is  no  end-to-end  level  retransmission  -  this  being  done  at  the  Gatt?way 
level.  The  analysis  of  Appendix  1  ^plies  to  each  PAR  stage,  with  the 
parameters  of  the  simulation  model  (S^tion  5.2)  substituting  for  corresponding 
parameters  in  Appendix  1.  The  average  inter-packet  arrival  interval  A^^j  at 
stage  i+1  is  given  by: 

Aj^^j  *  Ai/[(1  -  Pi)  X  (1  +  av.  retransmissions  per  packet  in 
stage  i  (eq'9))] 

with  Aj  -  A  (*  av.  inter-message  arrival 
interval.  Section  5.2) 

=  Packet  loss  probability  in  the 
i'th  net  (Section  5.2) 

The  above  expression  takes  into  account  the  production  of  retransmission 
duplicates  by  each  PAR  stage,  and  allows  the  determination  of  the  average 
retransmission  queue  size  (eq'lO)  and  the  average  nurber  of  message  duplicates 
for  each  PAR  stage  (which  for  the  i'th  stage  is:  A/A^  x  retransmission 
overhead  for  the  i'th  stage,  from  eq'9). 

The  overall  end-to-end  delay,  and  the  average  nurber  of  packets  being  re- 
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sequenced  at  the  Destination  Host  (by  the  end-to-end  SPAR  protocol),  may  be 
obtained  reasonable  sinply,  if  the  receiving  Gate-Half  of  each  PAR  stage  is 
assuned  to  filter  out  (retransmission)  packet  diqalicates  (giving;  Aj=A,  for 
all  i).  The  end-to-end  delay  derived  thus,  will  be  slightly  larger  than 
without  such  diqplicate  filtering,  since  message  duplicates  increase  the  chance 
of  early  arrival  at  the  Destination.  However,  for  low  packet  loss  (and  hence 
low  duplicate  production),  these  results  serve  for  model  validation.  The 
probability  density  function  (pdf)  gn,  and  probability  distributicMi  function 
(PDF)  CN  for  the  end-to-end  delay  through  stages  1  to  n,  are  given  by  the 
convolution  of  the  pdf's  for  (one-delay)  delay  across  each  PAR  stage: 

^  *1  \  2 
gn(x)  »  rgi(x-Xj)J  g2(Xl-X2)  •••  Jjj  ‘  *r-l<V2-Vl>- 

«n‘Vl>-*S,-l‘'=‘n-2  •••  “'*1 

X 

(24(x)  =  r  gn(u).du 
'^0 

with  gj  *  g  from  eq'5  (Appendix  1)  for  the 
i ' th  stage 

The  following  performance  measures  can  be  obtained  from  GN: 


eo 


PDF  of  end-to-end  delay 
with  sequencing 

=  HN(x) 

oo 

=  n’CN(x+j.A) 

J=o 

(13) 

av.  unsequenced  end-to- 
end  delay 

=  Ju- 
'^0 

GN(x)).dx 

(14) 

av.  sequenced  end-to-end 
delay 

oo 

=  r  (1  - 
•^0 

HN(x)).dx 

(15) 

av.  no.  packets  being 
sequenced  at  Destination 

=  (av.  sequenced  end-to-«id  delay 
-  av.  unsequenced  end-to-end 

(16) 

delay) /A 


with  A  *  inter-message  arrival  interval 
(Section  5.2) 

Ife  note  that  equation  13  ejqjresses  the  requirement  that  for  a  packet  to  be  in 
sequence  at  the  Destination,  it  and  all  its  predecessors  must  have  arrived. 
Equation  16  uses  Little's  result  (Little  61)  for  average  customer  queue  size, 
given  the  average  queueing  delay  and  average  customer  arrival  rate. 

We  now  look  at  the  protocol  mix  Nil( SPAR, SPAR,  ...  ,  SPAR)  -  used  for  the 
Hop-by-Hop  approach  in  Section  6.  Here  the  average  inter-packet  arrival 
interval  at  each  stage  is  A,  since  each  SPAR  stage  filters  out  duplicates. 

The  performance  for  the  first  stage  can  be  obtained  from  Appendix  1,  since 
for  the  first  stage  (only),  packets  arrive  at  constant  intervals.  Performance 
for  the  other  stages  can  be  obtained  approximately  ( from  Appendix  1 ),  by 
assuniiv  that  packet  arrivals  at  these  are  also  at  constant  intervals.  The 
average  end-to-end  message  delay  is  given  approximately  by  the  sun  of  the 
delays  for  each  stage,  obtained  in  the  above  way.  Such  jq^roximate  results 
serve  to  partially  validate  corresponding  simulation  results. 


7.3  A  Survey  of  End-to-End  Retransmission  Techniques 
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1,  Introduction 

Retransmission  of  lost  or  damaged  messages  from  a  sender 
to  a  receiver  is  a  basic  ingredient  of  computer  network 
protocols.  It  can  occur  at  many  levels  from  a  simple 
point-to-point  line  level  to  an  end-to-end  connection 
across  a  number  of  levels  of  data  path,  where  several 
mechanisms  combine  to  ensure  reliable  data  transfer. 

Specific  instances  of  retransmission  are  found  at  line 
level  in  protocols  such  as  HDLC  (CCITT  76),  and  between 
packet  switches  in  a  network  (McQuillan  77).  In  a  multiple 
network  environment,  retransmission  might  be  employed  between 
gateways  operating  at  the  edges  of  the  individual  networks 
(Sunshine  77a).  Finally,  a  reliabledelivery  end-to-end 
protocol  may  support  process-process  communication  across  a 
one  -  or  many  -  network  path ,  and  such  a  protocol  may  also 
require  a  retransmission  capability.  Examples  of  the  latter 
are:  TCP  (Cerf  78a),  INWG  96  (Cerf  78b),  the  EIN  end-to-end 
protocol  (EIN  76),  and  the  CYCLADES  end-to-end  protocol 
(CYCLADES  73). 

Other  instances  of  retransmission  occur  in  packet  switching 
over  a  broadcast  medium,  and  specialised  retransmission  schemes 
have  evolved  for  use  in  broadcast  satellite  operation 
(Binder  75),  packet  radio  (Kunzelman  78),  and  with  Ethernet 
(Metcalfe  76). 

This  paper  concentrates  on  the  aims  of  end-to-end  retransmission. 
In  particular  we  look  at  two  alternative  schemes:  positive 
acknowledgement  of  data  coupled  to  a  sender  timeout,  which  is 
well-researched,  and  use  of  negative  acknowledgement,  which 
is  not.  These  are  reviewed  in  terms  of  a  defined  set  of 
retransmission  objectives,  and  the  advantages  and  disadvantages 
of  each  scheme  are  demonstrated  using  a  simulation  model  of 
a  simple  end-to-end  connection. 
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We  note  that  in  our  terminology,  a  "transport  station" 
will  denote  the  physical  implementation  of  an  end-to- 
end  protocol  at  a  particular  site,  and  end-to-end 
communication  will  be  between  pairs  of  "processes". 

2.  Aims  of  End-to-End  Retransmission 

The  aims  of  a  retransmission  mechanism  are: 

ensuring  reliability 

minimising  delay 

minimising  redundant  duplicate  retransmissions 

simple  operation 

These  aims  relate  to  any  level  of  protocol,  but  particularly 
at  the  end-to-end  level,  where  both  the  delay  and  variation 
in  delay  are  scaled  up  to  such  a  degree  that  meeting  the 
criteria  mentioned  above  becomes  rather  more  of  a  critical 
matter. 

The  minimisation  of  delay  is  important  in  packet  switched 
networks,  where  nodal  switching  delays  add  up  to  a 
significant  amount  compared  to  say  the  delay  in  a  pre- 
established  digital  circuit.  For  example,  the  transit  delay 
of  existing  X25  networks  is  typically  one  sixth  to  half  a 
second  (Erskine  77,  Guilbert  77).  Howevever,  retransmissions, 
which  will  normally  be  keyed  to  some  message  (or  lack  of  it) 
from  the  receiver,  are  likely  to  be  delayed  by  at  least 
two  such  intervals. 

Minimisation  of  retransmission  overhead  is  an  obvious  criteria 
related  to  efficient  use  of  resources  -  both  communication 
resources  to  carry  the  unnecessary  additional  messages,  and 
processor  resources  in  generating  and  interpreting  the 
messages.  In  public  networks,  packet  costs  will  be  incurred 
for  such  messages,  and  in  large  private  datagram  networks, 
runaway  retransmissions  may  cause  network  congestion,  which 
is  not  easily  recoverable.  We  note  that  efficient  resource 
utilisation  and  minimisation  of  delay  are  also  the  goals 
of  flow  control  (Pouzin  76). 

Finally,  simplicity  is  important,  both  to  allow  unambiguous 
definition  and  to  reduce  the  size  and  complexity  of  transport 
stations . 
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The  diversity  of  these  aims  arpues  against  a  single  scheme 
for  all  situations.  By  broadly  classifying  different  end- 
to-end  reliability  levels  it  can  be  shown,  however,  that 
schemes  can  be  introduced  to  satisfy  the  different  require¬ 
ments. 

3.  Positive  Acknowledgement  Retransmission 

Positive  acknowledgement  retransmission,  using  a  timeout  at 
the  sender,  is  the  most  common  retransmission  scheme  adopted. 

Its  main  advantage  is  simplicity;  data  made  available  to  a 
receiving  process  is  positively  acknowledged  by  the  receiving 
station;  data  which  times  out  at  a  sending  station  is  retrans¬ 
mitted.  Reliability  is  virtually  guaranteed  in  all  circum¬ 
stances  short  of  system  crashes,  which  might  permanently 
remove  the  necessary  state  information  to  maintain  the  connection 
(Sunshine  75).  In  addition,  however,  careful  re-use  of 
either  message  sequence  numbers  (Tomlinson  74,  Dalai  74)  or 
connection  identifiers  (Reed  77)  will  be  necessary,  in  order 
to  reliably  detect  message  arrivals  from  previous  (recent) 
incarnations  of  a  connection. 

Analysis  (Sunshine  75)  has  shown  the  delay  incurred  by  the 
need  to  rely,  on  occasions,  on  retransmitted  packets,  is 
reduced  by  using  a  smaller  retransmission  timeout.  Minimum 
retransmissions,  however,  are  achieved  by  employing  a  timeout 
equal  to  twice  the  maximum  packet  life  time  for  the  communi¬ 
cation  path  (if  known),  thus  ensuring  that  retransmissions  are 
only  triggered  when  a  packet  is  definitely  lost  or  damaged. 

Since  the  retransmission  delay  with  such  a  timeout  could  be 
excessive,  a  smaller  "tuned"  timeout  might  be  used,  where  the 
probability  of  retransmitting  a  packet,  about  to  be  acknowledged, 
is  kept  very  small.  The  value  of  such  a  tuned  timeout  will 
always  exceed  the  average  round  trip  delay  of  the  transit 
medium  -  to  avoid  runaway  retransmission  -  but  the  excess  could 
be  small  (Sunshine  75). 

The  most  serious  drawback  of  positive  acknowledgement  re¬ 
transmission  is  that,  under  certain  conditions,  a  very  high 
level  of  redundant  duplicate  retransmission  is  unavoidable 
(McKenzie  74).  This  is  possible  whenever  two  or  more  data 
carrying  packets  are  pipelined  (i.e.  simultaneously  out¬ 
standing)  at  a  sender.  Because  these  must  be  acknowledged  in 
sequence  (following  data  delivery  to  a  receive  process)  the 
loss  of  a  single  packet  delays  the  acknowledgement  of  all 
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subsequent  packets  and  unnecessarily  induces  their  retrans^ 
mission.  The  pipelining  of  a  large  number  of  packets  may 
thus  lead  to  a  large  ratio  of  retransmitted  packets  to 
packets  actually  lost.  Further,  in  schemes  where  a  large 
unit  of  acknowledgement  is  used  -  such  as  a  letter  in  INWG 
96  -  unnecessary  retransmission  will  be  even  higher,  since  in 
addition  to  succeeding  jiackets,  those  packets  preceeding  a 
damaged  packet,  but  belonging  to  the  same  acknowledgement 
unit,  will  also  require  retransmission. 

The  seriousness  of  the  above  obviously  relates  to  the  level 
of  end-to-end  packet  loss.  For  very  low  loss  rates,  even 
when  acknowledgement  is  per  letter,  the  absolute  magnitude 
of  retransmission  is  likely  to  be  low  (Day  75).  However, 
for  high  loss  rates,  alternative  retransmission  schemes  may 
be  needed,  which  reduce  the  level  of  redundant  retransmission. 

3.1  Modifications  of  Positive  Acknowledgement  Retransmission 

We  explore  several  modifications  to  positive  acknowledgement 
retransmissions,  which  reduce  the  number  of  redundant  re¬ 
transmissions. 

Several  authors  (McKenzie  74,  Sunshine  75)  have  suggested 
retransmitting  only  the  first  packet  timed  out  (repeatedly 
if  necessary)  on  the  assumption  that  for  a  low  end-to-end 
loss  rate,  this  will  probably  be  the  only  packet  lost.  The 
main  disadvantage  of  this  scheme  is  that  it  would  have  a 
very  high  recovery  delay  whenever  a  large  number  of  packets 
were  discarded  at  a  receiving  station  -  the  occurrence  of 
which  can  be  an  important  reason  for  requiring  an  end-to- 
end  retransmission  capability  (Cerf  74).  We  therefore 
conclude  that  this  is  not  a  good  general  scheme. 

In  another  scheme,  which  has  been  proposed  for  TCP 
(Mathis  77),  the  retransmission  interval  for  each  packet 
commences  at  some  base  value  and  increases  linearly  or 
exponentially  following  each  retransmission  of  the  packet. 

The  main  purpose  is  to  avoid  flooding  the  subnet  and  receiver 
with  retransmissions,  in  the  event  that  packets  have  to 
be  discarded  at  the  receiver  (or  in  the  subnet)  and  flow 
control  alone  is  inadequate  to  quench  this  flow.  We  expect 
this  scheme  will  be  most  useful  with  transport  stations 
operating  over  datagram  networks  which  implement  minimal 
congestion  control.  It  is  interesting  to  note  that  an 
increasing  retransmission  interval  may  also  be  used  in  broad¬ 
cast  networks  for  a  similar  purpose  (Metcalfe  73). 
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As  we  noted  above,  the  retransmission  timeout  used  with 
positive  acknowledgement  is  subject  to  two  constraints: 
it  should  be  large  enough  to  minimise,  or  at  least 
considerably  reduce,  the  probability  of  premature  re¬ 
transmission  and  it  should  avoid  unnecessarily  high  re¬ 
transmission  delays.  The  importance  of  the  latter  is 
obviously  related  to  the  frequency  with  which  retransmission 
is  required.  Thus  for  extremely  reliable  end-to-end  paths, 
such  as  occur  across  the  Arpanet  or  across  X25  nets, 
a  single  large  timeout  might  be  used  at  a  transport  station, 
which  can  be  pre-set  to  minimise  retransmission  on  all 
connections.  For  less  reliable  paths,  such  a  large  timeout 
would  be  unsuitable  for  those  connections  with  a  low 
transit  delay.  Instead,  connections  could  be  partitioned 
into  categories  -  e.g.  satellite,  many  hop  terrestrial,  etc. 

-  based  upon  their  order  of  round  trip  time,  with  a  suitable 
timeout  for  each  category.  Finally,  for  very  unreliable 
paths,  the  retransmission  timeout  might  have  to  be  tuned 
individually  to  each  connection,  in  order  to  minimise 
retransmission  delay.  The  following  algorithm,  which, 
continuously  re-evaluates  the  retransmission  timeout  from 
round  trip  delay  measurements,  is  one  method  of  achieving 
this. 

Each  time  that  a  new  packet  is  acknowledged  at  a  sending 
transport  station,  the  round  trip  delay  t  since  it  was  first 
transmitted  is  determined,  using  timestamping  information 
associated  with  the  packet  retransmission  queue.  Then, 
provided  the  packet  was  not  retransmitted  -  when  t  could 
be  misleading  -  the  average  round  trip  delay  estimate  T, 
for  the  connection,  is  updated  as  follows: 

T  ;=  (m.T  +  t)/(m+l)  m>=o  (1) 

The  value  of  T  obtained  thus  is  affected  by  all  the  delays 
inherent  to  the  connection,  and  is  consequently  more  useful 
than  knowledge  of  the  delay  of  the  transit  medium  alone. 
Appendix  1  shows  that  the  weight  m  has  two  features :  reduced 
m  reduces  the  delay  in  correcting  T  when  the  connection 
round  trip  delay  changes;  but  increased  m  reduces  statistical 
fluctuation  of  T  in  steady  state. 
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The  retransmission  interval  can  be  periodically  updated 
from  the  latest  value  of  T: 

Retransmission  Timeout  :=  n.T 

The  multiplier  n,  which  should  exceed  1,  is  set  to 
ensure  a  low  probability  of  premature  retransmission. 

Its  setting  will  thus  depend  on  the  likely  fluctuation 
in  acknowledg€>ment  delay,  and  could  be  small  for  low 
f luctuat ion . 

Negative  Acknowledgement  Retransmission 

Retransmission  induced  by  explicit  negative  acknowledgement 
of  lost  packets  by  a  receiver  is  far  less  common,  at  all 
levels,  than  positive  acknowledgement  retransmission.  One 
example  where  it  is  used  is  in  HDLC  (CCITT  76),  which  uses 
two  types  of  reject  command  to  prompt  retransmission 
(Gelenbe  78).  In  the  end-to-end  case,  there  is  a  proposal 
to  incorporate  negative  acknowledgement  in  the  INWG  96 
protocol  (Cerf  78b). 

The  major  disadvantage  of  using  negative  acknowledgement  in 
end-to-end  protocols  is  the  comple.xity  this  would  add.  For 
example,  a  receiving  transoort  station  would  have  to  detect 
packets  lost  en  route,  possibly  by  timing-out  missing 
message  fragments.  Furthermore,  positive  acknowledgement 
retransmission,  with  a  suitably  large  timeout  (Pouzin  73), 
would  still  be  required  to  ensure  reliability,  in  the  event 
that  negative  acknowledgements  or  retransmissions  were  lost. 
Positive  acknowledgement  retransmission  would  probably  also 
be  required  for  interactive  traffic,  where  loss  of  a  small 
isolated  message  might  go  undetected  at  a  receiving  transport 
station.  In  the  latter  case,  use  of  a  small  sender  timeout 
to  hasten  positive  acknowledgement  retransmission  could  be 
integrated  with  negative  acknowledgement  by  restarting  the 
timeout  of  a  packet,  whenever  it  or  a  packet  with  lower 
sequence  than  itself  was  negatively  acknowledged. 

The  main  advantage  of  negative  acknowledgement  retransmission 
is  to  reduce  redundant  retransmission.  For  example, 
negative  acknowledgement  of  missing  "letter"  fragments 
(see  Section  5)  might  be  used  in  INWG  96,  in  addition  to 
positive  acknowledgement  of  whole  letters.  With  pipelined 
traffic,  this  additional  means  of  re-supplying  lost  packets 
would  substantially  reduce  the  likelihood  (discussed  in 
Section  3)  of  unnecessarily  retransmitting  successors  to  - 
or  packets  in  the  same  letter  as  -  a  damaged  packet.  As  an 


alternative,  the  unit  of  positive  acknowledgement  in 
INWG  96  could  be  reduced  -  e.g.  to  letter  fragments  - 
to  avoid  retransmitting  a  whole  letter  whenever  a 
letter  portion  was  lost,  but  this  would  not  avoid  re¬ 
transmitting  packet  successors  when  traffic  was  pipelined. 


5.  Comparison  of  Positive  and  Negative  Acknowledgement 
Retransmission 

We  illustrate  the  points  made  above  about  each  basic 
retransmission  scheme  with  some  simulation  and  analytical 
results  for  an  example  model  of  an  end-to-end  connection. 
The  model,  we  will  describe,  comprises  a  single  process- 
to-process  connection  maintained  by  a  pair  of  transport 
stations  (TS's),  where  data  flow  is  one  way  from  a  Send 
Process  (source)  to  a  Receive  Process  (sink).  The  data 
transport  mechanism,  under  review,  is  a  simple  one  common 
to  many  end-to-end  protocols,  such  as  TCP(Cerf  77)  and  INWG 
96  (Cert  78b).  Figure  1  illustrates  the  connection. 
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An  infinite  stream  of  fixed  size  "letters"  Is  passed  one 
at  a  time  and  at  constant  time  intervals  from  the  Send 
Process  to  the  Sending  TS.  Each  letter  is  encapsulated 
in  a  single  packet,  which  is  assigned  a  monotonical ly 
increasing  letter  sequence  number,  and  transmitted  immediate¬ 
ly.  There  is  no  restriction  on  flow,  corresponding  to  a 
very  large  window  and  unlimited  internal  buffering  in  the 
case  of  TCP  and  INWG  96.  Further,  packets  are  not  fragmented 
(e.g.  at  gateways)  during  transit.  The  transit  delay 
between  the  two  TS's  has  an  Erlang  distribution  with 
parameter  k  and  mean  1/u  (which  we  set  to  unity).  The 
probability  density  function  f  for  a  delay  x  is: 

f(x)  -  (k.u)‘^.x‘^“^e‘*‘^*/(k-l):  x>-0  (2) 

This  models  a  wide  range  of  delay  distributions,  from 
exponential  (k=l)to  constant  (k  approaches  infinity). 

The  coefficient  of  variation  of  delay  is  k~l.  We  will  use 
two  values  of  k,  k»25  and  k»4 ,  to  model  low  and  high  delay 
variability  respectively.  Loss  of  packets  in  the  transit 
medium  (or  at  the  receiving  TS)  is  represented  by  applying 
a  fixed  loss  probability  to  each  packet  in  transit  independent 
ly.  The  delay  distribution  and  loss  are  identical  for 
both  data  packets  and  reverse  travelling  acknowledgements. 

The  receiving  TS  buffers  out  of  order  letter  arrivals, 
and  whenever  possible,  passes  letters  in  sequence  to  the 
receive  process  -  which  can  always  accept  them.  Further, 
a  single  acknowledgement  packet  is  returned  immediately  to 
the  sending  TS ,  for  all  letters  delivered  at  the  same  time 
(thisalso  re-acknowledges  all  earlier  letter  deliveries) 
and  carries  the  sequence  number  of  the  next  letter  to  be 
delivered.  Such  an  acknowledgement  is  also  returned  for 
each  duplicate  letter  arrival;  the  latter  then  being 
discarded.  At  the  Send  TS ,  unacknowledged  letters  are 
retransmitted  after  a  constant  timeout  interval,  since  their 
previous  (re)transmission ,  and  they  are  discarded  upon 
acknowledgement.  Internal  TS  processing  delays  are  ignored, 
but  may  be  regarded  as  comprising  part  of  the  transit  medium 
delay. 

The  unit  of  time  is  the  mean  TS-TS  transit  delay  (1/u  in  eq.2) 
The  model  than  has  the  following  parameters: 

k:  parameter  of  Erlang  delay  distribution 

A:  inter-letter  arrival  Interval  at  the  send  TS 

p:  probability  of  packet  loss  in  the  transit  medium 

R;  send  TS  retransmission  interval 
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The  model  described  so  far  is  used  to  illustrate  positive 
acknowledgement  retransmission.  We  illustrate  negative 
acknowledgement  with  the  following  additional  mechanism, 
which  has  been  chosen  for  its  simplicity.  The  Receive  TS 
buffers  and  delivers  arriving  letters  as  before.  However, 
each  arriving  letter  commences  a  timeout  period  if  its 
immediate  sequential  predecessor  has  not  yet  arrived.  If 
the  latter  has  not  arrived  on  timeout,  a  negative  acknowledge¬ 
ment  is  returned  to  the  Send  TS  for  all  the  immediately 
preceding  non-arrivals  -  up  to  but  not  including  the  highest 
sequence  preceding  letter  which  has  arrived.  In  practice, 
only  the  highest  and  lowest  sequence  numbers  of  these  would  be 
physically  carried.  Negative  acknowledgements  are  also 
lost  with  probability  p,  representing  say  the  case  where 
acknowledgements  are  piggy-backed  in  a  reverse  data  stream. 

At  the  Send  TS ,  negatively  acknowledged  letters  are  immediate¬ 
ly  retransmitted,  independently  of  positive  acknowledgement 
retransmission  (which  operates  as  before).  When  implemented 
this  scheme  requires  a  new  parameter: 

Tnack  =  Timeout  at  the  receive  TS 

The  first  example  we  illustrate  with  this  model  concerns  the 
average  number  of  times  letters  are  retransmitted  with 
positive  acknowledgement  retransmission  operating  only-, 
for  different  values  of  the  retransmission  timeout  R. 
Simulation  results  for  this  are  shown  in  Fig. 2,  with  A  set 
to  represent  real-time  on  bulk  transfer  traffic.  The 
first  point  to  notice  is  that  for  a  large  enough  R,  re¬ 
transmission  is  minimised  for  each  case  considered,  as  we 
stated  earlier.  Tuned  R  -  discussed  in  Section  3  -  is  the 
minimum  value  of  R  giving  this  minimisation,  and  we  will  call 
this  value  Rtun.  It  clearly  exceeds  the  average  normalised 
round  trip  delay  (which  is  2)  in  every  case,  and  is  increased 
by  higher  delay  variability  (reduced  k).  R  less  than  Rtun 
causes  premature  letter  retransmission,  and  the  rise  of  this 
(with  reduced  R)  becomes  more  stepwise  as  acknowledgement 
delay  approaches  a  constant  (k  approaches  infinity,  p 
approaches  zero).  Excessive  retransmission  for  pipelined 
traffic  -  discussed  in  Section  3  -  is  illustrated  for  the 
case  k=25,  p=l ,  where  the  minimum  level  of  retransmission  at 
.54  substantially  exceeds  the  minimum  requirement  of  .11, 
when  only  lost  packets  are  retransmitted. 

Next  we  look  at  this  situation  from  the  i>oint  of  view  of 
negative  acknowledgement.  Figure  3  shows  simulation  results 
relating  the  value  of  the  timeout  Tnack  used  at  the  receiver 
to  the  extent  of  redundant  retransmission  when  there  is  no 
packet  loss  (p“0).  The  results  are  analogous  to  those  in 
Fig. 2.  A  low  value  of  Tnack  increases  the  likelihood  of  pre¬ 
maturely  negatively  acknowledging  letters,  which  have  not 
arrived,  thereby  producing  redundant  retransmissions. 


9 


Conversely,  for  large  enougn  Tnack,  redundant  retransmission 
ceases.  We  can  thus  determine  a  tuned  Tnack  value  to 
reduce  the  retransmission  delay  when  packets  are  lost,  in 
the  same  way  that  Rtun  was  found  above.  Figpire  3  shows  that 
its  value  depends  on  the  variability  of  the  transit  medium 
delay  (k)  and  the  traffic  rate  (A),  each  of  which  affects 
the  extent  to  which  letters  can  arrive  out  of  order. 


R 

Figure  2.  Average  Retransmlssione  per  Letter  vereus  Sender  Tiaeout 
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Floure  3.  Average  Retranamlsalone  per  Letter  vereui  Receiver  Timeout 
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The  proceeding  results  allow  comparison  ot  the  minimum  (tuned) 
retransmission  delay  inherent  in  either  scheme.  For  positive 
acknowledgement,  this  is  simply  Rtun.  For  negative  acknow¬ 


ledgement,  an  approximate  retransmission  delay  is  obtained 
by  assuming  that  the  immediate  sequential  successor  to  a 
lost  letter  arrives  safely  at  the  receiver,  and  after  timeout 
and  return  of  a  negative  acknowledgement,  induces  the  retrans¬ 
mission  (l.e.  p  is  small,  k  is  large).  Thus: 

av.  retransmission  -  av.  round  trip  Tnack  ♦  T 
delay  delay 

-  2  ♦  Tnack  ♦  T 

For  the  case  pK),  k>25.  A". 4,  Rtun  is  3  (Fig.  2),  giving 
equality  of  retransmission  delay  for  tuned  Tnack  •■.6  (Fig\  3). 
The  case  p-0,  k-4,  A-.4  is  similar,  providing  near  equality. 


riyur*  4.  Avarag*  l.attar  Dalay  varaua  Packat  Loaa 
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The  above  comparison  implies  that  delays  in  either  scheme 
should  be  similar.  We  investigate  this  in  Fig. 4,  which  shows 
the  average  letter  delay,  from  the  Send  to  the  Receive  Process, 
against  packet  loss.  Simulation  results  are  shown  for 
negative  acknowledgement,  whereas  for  positive  acknowledgement 
we  use  analytical  results,  from  Appendix  2,  whose  accuracy 
slightly  exceeds  that  obtained  from  simulation.  We  re-use 
the  tuned  timeouts  quoted  above,  so  it  is  not  surprising  to 
find  in  Fig. 4  almost  identical  delays  for  a  low  loss  rate. 

At  higher  loss  rates,  the  delay  for  negative  acknowledgement 
becomes  much  larger,  because  substantially  more  lost  packets 
require  positive  acknowledgement  retransmission,  following 
the  loss  of  a  negative  acknowledgement  or  retransmission, 
and  this  uses  a  large  untuned  timeout  (R*10).  The  latter 
setting  reflects  the  reduced  significance  of  the  sender  time¬ 
out  with  negative  acknow-ledgement  retransmission,  although  we 
would  obviously  expect  to  reduce  delay  by  employing  a  smaller 
sender  timeout. 


p  (log  aeolo) 


Plguro  5.  Avorago  Mtranamlssiona  por  Lottor  varaua  Packet  toaa 


In  Fig.  5,  we  compare  the  retransmission  overhead  of  each 
scheme  -  obtained  by  simulation  -  over  the  same  range  of  loss 
rates  used  in  Fig.  4.  For  low  loss  rates,  retransmission 
in  either  scheme  is  low.  However,  it  is  clear  that  negative 
acknowledgement  produces  far  fewer  retransmissions  than 
positive  acknowledgement,  and  the  magnitude  of  the  difference 
becomes  more  significant  for  increased  packet  loss. 


6.  Selection  of  a  Retransmission  Scheme 

We  evaluate  here  the  two  basic  schemes  we  have  been  comparing, 
in  the  context  of  three  levels  of  end-to-end  reliability. 


6.1.  Highly  Reliable  End-to-End  Path 

In  this  case  we  assume  that  lower  level  mechanisms  provide 
reasonably  reliable  communication.  An  X25  virtual  call 
netwoifk  is  an  obvious_example,  where  the  bit  error  probability 
can  be  as  low  as  10“ (Danet  76).  Such  a  loss  is  acceptable 
for  most  network  use,  but  where  a  guaranteed  process- to- 
process  reliability  several  orders  of  magnitude  better  is 
required,  we  may  expect  to  superimpose  an  additional  reliable 
end-to-end  protocol.  As  an  example,  a  version  of  the  INWG 
96  protocol  adopted  specifically  for  use  above  X25  networks 
is  currently  being  produced  by  an  IFIP  working  group  (Cerf  78b). 

It  is  clear  that  the  frequency  of  necessary  retransmission  is 
sufficiently  low  that  positive  acknowledgement  retransmission 
using  a  long  timeout,  will  be  adequate. 


6.2.  Moderately  Reliable  End-to-End  Path 

We  reserve  this  case  for  end-to-end  packet  loss  probabilities 
lying  between  10~^  and  10**°,  the  range  likely  to  be  found  in 
most  datagram  networks,  or  where  a  receiving  transport  station 
may  occasionally  discard  packets  due  to  flow  control  (Cerf  74). 
Here  retransmission  will  be  required  too  infrequently  to  make 
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the  reduction  in  retransmission  overhead,  possible  with 
negative  acknowledgement,  worthwhile.  Positive  acknow¬ 
ledgement  will  thus  be  suitable,  and  the  value  of  the  time¬ 
out  may  be  set  by  categorising  connections,  as  discussed  in 
Section  3.2 


6.3.  Unreliable  End-to-End  Path 

-2 

This  case  is  for  packet  loss  probabilities  less  than  10 
Such  a  high  loss  rate  is  obviously  atypical,  because  in  nearly 
all  practical  situations,  lower  level  mechanisms  reduce  the 
loss  rate  seen  at  the  end-to-end  level.  However,  it  may 
occur  in  special  circumstances,  such  as  where  a  receiving 
station  uses  inadequate  buffering  and  must  frequently  discard 
arriving  packets  (Edge  77). 

Provided  a  high  level  of  retransmission  overhead  is  acceptable, 
positive  acknowledgement  retransmission  could  be  used  here, 
and  the  retransmission  timeout  would  probably  require  tuning  - 
as  described  in  Section  3.2  -  to  reduce  delay.  Alternatively, 
the  simulation  results  discussed  in  Section  5  show  that  a 
negative  acknowledgement  scheme  could  substantially  reduce 
the  level  of  unnecessary  retransmission  at  the  cost  of 
somewhat  larger  delay. 


7.  Conclusions 

We  conclude  that  positive  acknowledgement  retransmission  - 
as  employed  in  many  existing  end-to-end  protocols  -  is  most 
suitable  for  the  majority  of  end-to-end  connections,  namely 
those  with  moderate  or  high  reliability.  For  unreliable 
connections,  negative  acknowledgement  retransmission  may 
be  preferable,  because  this  greatly  reduces  the  extent  of 
redundant  retransmission,  which  can  be  large  with  just 
positive  acknowledgement.  The  drawbacks  to  negative  acknow¬ 
ledgement  are  added  complexity  and  probable  higher  delay. 
Optimal  performance  in  either  scheme  requires  careful 
selection  of  timeout  values,  and  for  unreliable  connections, 
these  should  be  tuned  to  the  Individual  connection. 
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APPENDIX  1  -  Analysis  of  an  Algorithm  to  Estimate  Round 
Trip  Time 

The  algorithm  updates  the  current  round  trip  delay  estimate 
T  ,  lor  a  connection,  with  successive  round  trip  measurements 
t°  according  to  either  of  the  following  equivalent  expressions: 

’'n+1  t^)/(m+l)  m>-  o  (3) 

-  ♦  (VT^)/(m+l)  (4) 

If  the  actual  round  trip  delay  changes  suddenly,  and  the 
latest  measurement  t  is  a  better  estimate  than  T  ,  then  Eq.4 
shows  T  moves  closer^to  the  more  correct  value  for  smaller  m. 
Conversely,  in  steady  state,  after  a  sequence  tQ,t^,  ...  ,tQ 
of  updates,  is  given  by  (using  Eq.3): 

»  (m+l)"^.(tjj  +  m(m+l)~^t^_j  +  m^(m+l)^t^_2  ♦  ••• 

♦  m“(m+l)~“tQ  +  m“*^(m+l)"^°'^^^TQ) 

If  we  regard  each  t.  as  a  random  variable,  and  make  the 
approximation  that  ^they  are  independent  and  identically 
distributed  (whence  the  steady  state  assximption) ,  then  simple 
expressions  lor  the  expectation  (E)  and  variance  (var)  of  T^^j^ 
can  be  obtained  for  the  case  n  approaches  infinity: 

E(T^^j)  »  E(t^)  for  each  i 

var  »  (m+l)"^.(var(t^)  +  m^(m+l)"^var(t^_j^)  + 

m^(mi-l)”^var(tj^_2) 

“  var( t^ )/(2m+l)  for  each  i 

The  last  expression  above  shows  that  statistical  fluctuation 
of  T„  is  reduced  for  larger  m. 


APPENDIX  2  -  Average  Letter  Delay  with  Positive  Acknowledgement 
Retransmission 

We  use  previous  analytical  study  (Sunshine  .75  and  Sunshine  77b) 
to  calculate  the  average  letter  delay  lor  th'e  connection  model 
defined  in  Section  5.  The  density  function  f  lor  letter  transit 
delay  (x)  is,  from  Eq.2,  Section  5: 
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f(x)  -  (k.u)‘‘.x^’^.e"‘‘'^*/(k-l):  x>-0 


The  probability  distribution  F  is: 

kiiv  i 

F(x)  -  1  -  (kux)J/J’.  x>-0 

j-O 

We  now  add  in  the  effects,  successively,  of  the  subnet  loss 
probability  p,  the  retransmission  interval  R,  and  the 
requirement  of  sequenced  letter  delivery.  The  probability  F' 
of  letter  arrival  after  time  x,  with  loss,  is: 

F'(x)  -  (l-p).F(x) 

The  probability  G  of  arrival,  with  retransmissions,  is: 

n 

G(x)  -  1  -^(1  -  F'(x-jR))  ;  n-tx/Rj 

The  probability  H  of  arrival  with  all  predecessors  present  is: 

H(x)  -  ^G(x+JA)  ;  A  ■  inter  letter 

j*0  arrival  time 


H(x) 


Average  letter  delay  is: 

average  letter  delay 


r<^- 


H(x)) .dx 
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VIII.  ARPANET  USAGE 


8.1  Organisational  Support.  Users  and  Future  Provision  of  Services 


There  was  a  significant  change  in  1978  in  the  way  ARPANET 
usage  was  regarded  in  the  UK.  Up  to  the  end  of  1977,  a  grant 
from  the  SRC  Computer  Science  Committee  had  covered  User 
Support  for  University  Users.  Since  that  time,  it  has  been 
decided  that  there  is  no  longer  a  research  element  in  the 
provision  of  the  service.  Instead,  the  Facilities  for 
Computing  Committee  of  the  Science  Research  Council  has 
authorised  an  agreement  to  provide  support  for  the  service. 
This  support  is,  therefore,  part  of  the  provision  of  computing 
services  to  UK  academic  research  workers. 


However,  the  Facilities  Committee  has  warned  that  it  may  well 
not  support  usage  after  1979.  We  are  exploring  whether  alter¬ 
native  communications  facilities  have  yet  become  available, 
e.g.  via  the  International  Packet  Switched  Service  (IPSS) 
and  Telenet .  The  EDUNET  Community  may  provide  such  an  alter¬ 
native  vehicle.  At  the  end  of  1978,  host  connection  of  UK 
Hosts  to  IPSS  was  not  yet  available.  Our  own  line  to  IPSS 
is  not  expected  before  April  1979.  Moreover,  normal  X25 
network  access  (which  UK  University  Hosts  will  use  to  the 
UK  PSS)  is  not  expected  before  October  1979.  Thus  it  is 
improbable  that  a  comprehensive  range  of  services,  like  File 
Transfer  and  widespread  local  interactive  terminal  support , 
would  be  possible  by  the  end  of  1979. 

The  users  outside  the  academic  community,  particularly  the 
Atomic  Energy  Authority  Culham  Laboratories  and  several 
Ministry  of  Defence  groups,  continue  to  desire  access.  In 
Culham' s  case,  the  access  requirement  is  dependent  on  the 
Itepartment  of  Enerjy's  Liveimore  Laboratories  not  being  available  via 
any  other  route.  They  have  indicated  that  they  would  like  contin¬ 
ued  access  at  least  up  to  the  end  of  the  first  quarter  of 
1980.  The  Ministry  of  Defence  is  organising  much  longer  term 
collaborative  activities  with  US  research  groups.  They  expect 
to  require  some  form  of  ARPANET  access  well  into  the  1980s., 
The  groups  authorised  at  the  end  of  1978  to  use  the  ARPANET 
link  are  listed  in  Table  8.1. 


8.2  Technical  Quality  of  Services 


The  actual  quality  of  the  services  provided  during  1978  was 
not  as  satisfactory  as  in  previous  years.  The  reasons  for 
the  poorer  quality  are  discussed  below.  Measures  have  been 
taken  to  overcome  the  deficiencies  when  they  are  under  our 
control,  and  a  much  better  service  is  expected  in  1979.  A 
number  of  unfortunate  service  interruptions  occurred  in  the 
second  half  of  1978.  These  were  due  mainly  to  the  effects 
of  industrial  disputes  in  the  Post  Office,  to  equipment 
problems  in  Norway  and  to  problems  with  our  own  air  condition¬ 
ing.  The  first  affected  the  length  of  time  required  to 
restore  service  in  the  event  of  line  failure.  The  second 
was  caused  by  several  incidents  of  lightning  damaging 


25. 


equipment  in  Norsar  during  vacation  periods.  Both  these 
occurrences  caused  total  isolation  from  the  US  for  several 
days  at  a  time.  The  third  forced  us  to  shut  down  the  PDF  9s, 
but  allowed  the  TIP  character  terminal  service  over  the 
switched  public  telephone  service  to  be  maintained. 

The  new  release  of  PDP  9  operating  and  network  systems  in 
mid-1978  allowed  access  via  EPSS  to  be  extended  to  an  essen¬ 
tially  24  hours  a  day  service.  The  usage  via  EPSS  on  a  regular 
basis  has  now  grown  significantly;  an  "E"  in  the  Usage 
Table  of  Table  8.1  indicates  access  via  EPSS.  Full  scale 
production  access  via  EPSS  brought  out  some  unfortunate 
mismatches  between  a  1 ine-at-a-time  system,  like  the  VPT 
character  support,  and  some  of  the  character  oriented 
application  programs  on  some  ARPANET  Hosts.  These  mismatches 
caused  particular  annoyances  with  multiple  echoes  of  some 
characters  (one  locally  and  one  from  ARPANET),  This  partial 
line  forwarding  was  facilitated  by  suitable  specific  action 
in  the  UCL  PDP  9  Gateway.  Annoying  consequences  could  not 
be  removed  completely,  however;  most  users  with  high  speed 
access  accepted  double  echoes  as  a  tolerable  way  of  working. 
One  reason  why  the  quantity  of  usages  via  EPSS  increased 
very  significantly  was  the  result  of  the  24  hoxrrs  a  day  access 
avaialbility .  Another  was  the  determined  effort  by  many  UK 
Hosts  to  improve  their  own  EPSS  support  in  preparation  for 
the  EPSS  Open  Day  in  June  -  mentioned  in  Section  4.5. 

The  provision  of  password  checking  and  the  quality  of  File 
Transfer  support  between  the  Rutherford  Laboratory 
IBM  360/195S  and  ARPANET  suffered  as  a  consequence  of  the 
introduction  of  the  new  release  of  the  operating  system. 

The  actual  operating  system  has  grown  so  large  that  the 
generation  of  a  new  version  is  dependent  on  the  availability 
of  a  larger  disk  than  the  standard  256  K  word  disk  used  for 
earlier  releases.  A  larger  40  M  byte  disk  was  brought  into 
service  in  1977.  It  has  not,  however,  proved  very  reliable. 
This  has  severely  hampered  system  development  of  the  PDP  9s, 
and  delayed  our  response  to  fixing  software  bugs.  By  the 
end  of  1978  most  of  the  deficiencies  had  been  identified 
and  remedied  in  our  development  system.  The  improvements 
were  scheduled  for  inclusion  in  the  Service  System  from 
the  first  quarter  of  1979.  Since  further  software  development 
planned  for  the  PDP  9s  is  minimal,  we  do  not  anticipate 
long-term  problems  in  development,  but  the  general  level 
of  PDP  9  reliability  will  clearly  worsen. 

Finally,  the  introduction  of  two  developments  on  ARPANET 
itself  have  increased  the  number  of  service  disconnections. 

One  development  is  changes  in  the  routing  algorithms  and 
other  standard  ARPANET  improvements.  The  UCL  mode  of  connect¬ 
ion  by  a  simple  two  hop  9.6Kbps  spur,  is  unique  on  ARPANET. 

For  this  reason  some  faults  appeared  only  when  new  releases 
were  put  in  the  field.  The  second  is  due  to  interference 
problems  when  one  tried  to  put  the  SATNET  circuit  as  an 
alternate  path.  Both  problems  are  expected  to  be  much 
alleviated  in  1979. 


ORGANISATION 

NAME 

ACCESS 

PROJECT 

SITE 

APPLETON  LABORATORY  (SRC) 

M.F.  REID 

INFRA-RED  ASTRONOMY 

R 

AMES,  RL 

ARCHAEOLOGY,  THE  INSTITUTE 
OF  LONDON  UNIVERSITY 

DR.  I.  GRAHAM 

ALGORITHM  DEVELOPMENT 

FOR  ANALYSING  ARCHAE¬ 
OLOGICAL  DATA 

RUTGERS- 10 
MIT-MC 

ASTON  UNIVERSITY 

D.  AVISON 

INVESTIGATION  OF  THE 
DATACOHPUTER 

CCA 

BIRMINGHAM  UNIVERSITY 

DR.  K.  LANG 

GUIDANCE  TO  REMOTE 
USERS  OF  COMPUTERS 

SRI,  CMU 

BLACXNEST  RESEARCH  EST. 

F.  GROVER 

C.  BLAMEY 

SEISMOLOGICAL  DATA 
EXCHANGE 

R 

ISI,  CCA,  1 
LLL 

BRISTOL  UNIVERSITY 

DR.  J.  ALCOCK 

DATA  ANALYSIS  ON 

ANTI -PROTON  SCATTER¬ 
ING 

R 

I£L,  CMU 

BRITISH  STEEL  CORP. 

D.  SHORTER 

HIGH  ORDER  LANGUAGE 
DEVELOPMENT  FOR 

US  DOD 

ISI 

CAMBRIDGE  UNIVERSITY 

PROF.  WILKES 

ALGEBRAIC  MANI¬ 
PULATION  SYSTEM 

E 

ISI,  MIT, 
UTAH 

CULHAM  LABORATORY 

R.  ENDSOR 

ALGOL  GENERATOR, 
ALGEBRAIC  SYSTEMS 

ILLIAC  IV, 
BBN  (for 
test) ,  ISI 

DURHAM  UNIVERSITY 

DR.  F.D.  GAULT 

EXCHANGE  OF  HIGH 
ENERGY  DATA  &  SOFT¬ 
WARE  DEVELOP. 

R 

LBL,  RL 

EDINBURGH  UNIVERSITY  (1) 

DR.  R.  BURSTALL 

PROGRAM  CXJRRECTED- 
NESS,  TRANSFORMATION 
&  SYNTHESIS 

ISIB,  UCLA 

EDINBURGH  UNIVERSITY  (2) 

M.  G(»DON 

PROOF  GENERATING 
SYSTEM  -  LCF 

SU-AI 

EDINBURGH  UNIVERSITY  (3) 

PROF.  D.  MICHIE 

ARTIFICIAL  INTELLI¬ 
GENCE  PR0<3UVMS 

ILLINOIS 

MYCIN 

ESSEX  UNIVERSITY 

DR.  J.M.  BRADY 

VISUAL  INFORMATION 
PROCESSING 

E 

MIT-AI 

HATFIELD  POLYTECHNIC  (1) 

DR.  G.M.  BULL 

BASIC 

E 

NBS 

HATFIELD  POLYTECHNIC  (2) 

DR.  A.V.  STOKES 

USER  INTERFACES  ON 
HETEROGENEOUS 

COMPUTER  NETWORKS 

E 

USC-ISI  & 
various 

IMPERIAL  COLLEGE 


DR.  J.  DARLINGTON 


PROGRAM  SYNTHESIS 


SRI-KL 
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NAME 

ACCESS 

ORGANISATION 

PROJECT  _ 

-  METHOD 

SITE 

KZNT  UNIVERSITY 

DR.  T.R.  HOPKINS 

ODEs,  PDEs  and  PAOE 
APPROXIMATIONS 

MIT-MC 

KINGS  COLLEGE, 

LONDON  UNIVERSITY 

N.D.  BIRRELL 

gUANTUM  FIELD  THEORY 
IN  CURVED  SPACE  TIME 

R 

MIT-MC  ,  RL 

LEEDS  UNIVERSITY 

DR.  A.P.  McCANN 

PORTABLE  SOFTWARE 

E 

NYU 

LIVERPOOL  UNIVERSITY 

P.  LENG 

ALGOL  68  DEVELOPMENT 

CMU 

MANCHESTER  UNIVERSITY 

DR.  R.N.  IBBETT 

MU 5  MODELLING  ALGOL 

63  COMPILER  DEVELOP¬ 
MENT 

E 

CMU 

MEDICAL  RESEARCH  COUNCIL 

DR.  R.T.  WILKINSON 

TELECONFERENCING  FOR 
NEUROPHYSIOLOGY  COLL¬ 
ABORATION  WITH 

DONCHIN  AT  BBN 

BBN 

MINISTRY  OF  DEFENCE 

D.  CURRY 

COLLABORATION  WITH 

US  ARMY  MATERIAL 
COMIAND  Hg 

ISI 

OFFICE- 1 

NAT.  PHYSICAL  lAB. 
(BARBER-EIN) 

MRS.  J.  ARMSTRONG 

PACKET  SWITCHED  NET¬ 
WORKS  PROTOCOLS 

E 

NAT.  PHYSICAL  LAB.  (2) 

L.A.  PINK 

INVEST.  THE  REgUIRE- 
MENTS  OF  UNIVERSAL 
NETWORK  ACCESS 
LANGUAGE 

E 

SRI-KL, 

MIT-AI 

NAT.  PHYSICAL  lAB.  (3) 

DR.  B.A.  WICHMAN 

PROGRAMMING  LANGUA¬ 
GES  (IRONMAN) 

E 

ISIA 

NEWCASTLE  UNIVERSITY  (1) 

A.J.  MASCALL 

SOFTWARE  DEVELOP .  TO 
ACCESS  EPSS  VIA  AN 
IBM-370  -  ASSISTING 
USERS  OF  ARPANET 

E 

various 

NEWCASTIX  UNIVERSITY  (2) 

J.  EVE 

ERROR  RECOVERY  IN 
COMPILERS 

E 

XEROX  PARC 

NORTH  IK)NDON  POLYTECHNIC 

DR.  A.P.  JOHNSON 

SYNTHESIS  DESIGN  OF 
C»GANIC  MOLECULES 

R 

HARVARD,  RL 

OPEN  UNIVERSITY 

DR.  M.  EISENSTAOT 

COMPUTATIONAL  MODEL¬ 
LING  OF  JOURNALISTIC 
TEXTS 

ISI 

OXFORD  UNIVERSITY  NUMER¬ 
ICAL  ALGORITHMS  GROUP  (1) 

S • J •  HAGUE 

NUMERICAL  SOFTWARE 

R 

ANL,  BBN,  H 

OXFORD  UNIVERSITY  (2) 

J.B.  MACALLISTER 

NUCLEAR  PHYSICS 

R 

HARVARD, 
ILLINOIS, 
BBN,  RL 

2b. 


ORGANISATION 
POST  OFFICE 
QUEEN  MARY  COLLEGE 

READING  UNIVERSITY  (1) 
READING  UNIVERSITY  (2) 

READING  UNIVERSITY  (3) 
ROYAL  COLLEGE  OF  ART 

ROYAL  HOLLOWAY  COLLEGE 

RSRE  (1) 

RSRE  (2) 

RUTHERFORD  LABORATORY 
SALFORD  UNIVERSITY 


NAME 

ACCESS 

PROJECT 

■  ■  -  METHOD 

SITE 

C.  BROOMFIELD 

SIMP  EXPERIMENTS 

BBN 

J.R.  HUTCHINSON 

DISTRIBUTED  COMP-  E 

UTER  SYSTEMS,  INTER¬ 
NETWORKING 

XEROX  PARC 
MIT-MULTICS 

PROF.  R.W.  HOCKNEY 

PARALLEL  COMPUTING  R 

ILLIAC  IV, RD 

OR.  R.H.  BERMAN 

DYNAMICS  OF  SPIRAL  R 

GALAXIES 

MIT,  RL 

DR.  D.  FINCHEM 

FUSED  SALTS  R 

ANL,  RL 

DR.  P.  PURCELL 

COMPUTER  AIDED 

DESIGN 

HARVARD 
MIT-MULTIC 
CMU,  UCLA 

DR.  R.H.  DAMERELL 

USE  OF  MACSYMA  FOR 

NUMBER  THEORY  APPLI¬ 
CATIONS 

MIT-MC 

DR.  J.M.  TAYLOR 

DR.  P.H.  MASTERMAN 

NETWORKING  E 

NELC,  MITRE 
RADC 

N.  NEVE 

SUPPORT  OF  US  F.VALU- 

ATIONS  OF  CORAL 

US  NAVY  LAB. 

MRS.  S.  WARD 

R 

RL 

DR.  G.  LAWS 

SUPERSONIC  FLOW  R 

PROBLEMS 

ILLIAC  IV,  RL 

SOUTHAMPTON  UNIVERSITY 
SYSTEM  RESEARCH  LIMITED 
THAMES  POLYTECHNIC 
UCL  (1) 

UCL  (2) 

UCH  MEDICAL  SCHOOL 

YORK  UNIVERSITY  (1) 

YORK  UNIVERSITY  (2) 


A.J.G.  HEY 

GAUGE  FIELD  THEORIES  R 

MIT-MC,  RL 

DR.  G.  PASK 

CONTRACT  WITH  US  AR^;V 

T.  CROWE 

NETWORK  EDITORS 

SRI 

PROF.  P.T.  KIRSTEIN 

PROBLEMS  IN  COMPUTER  E, 
NETWORK  DESIGN  AND  R 
APPLICATIONS 

various ,  RL 

T.  WESTGATE 

COMPUTER  CONFERENCING 

BBN 

DR.  L.  KOHOUT 

EVALUATION  OF  MEDICAL 
DATA  USING  GEDYS 

PACKAGE 

SUMEX-AIM 

PROF.  I.C.  PYLE 

CONFERENCE  WORD 
PROCESSING 

IDA,USC-ISIC 
ISIC,  ISI 

I.D.  COTTAM 

USE  OF  MODULA  FOR 
PROGRAMMING  SECURE 
OPERATING  SYSTEM 

KERNALS 

Note:  E  denotes  access  via  EPSS 
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IX.  STANDARDS  ACTIVITIES 

During  1978  significant  work  on  networking  standards  has 
been  done  by  CCITT  and  ISO,  involving  national  standards 
groups  in  all  countries  aware  of  the  need  for  standardiz¬ 
ation.  The  UCL  INDRA  group  has  participated  actively  in 
UK  deliberations,  for  instance  P.L.  Higginson  has  become 
rapporteur  to  two  of  the  ISO  working  groups  on  one  area 
of  the  model. 

The  thrust  of  the  ISO  work  has  been  to  develop  an  architect¬ 
ural  model  for  networking  consisting  of  7  layers  (34). 

The  familiar  level  to  those  involved  with  ARPA  TCP  working 
groups  is  level  4/5-  a  transport  station  interface  comparable 
to  the  interface  provided  by  the  TCP  protocol. 

The  model  also  puts  terminal  support  protocols  in  a  fixed 
part  of  the  model  (level  6),  and  file  transfer  protocols 
are  also  fitted  into  the  general  picture.  The  aim  of  the 
working  groups  is  to  take  existing  defined  protocols  and 
modify  them  to  fit  the  architectural  model. 

The  file  transfer  protocol-NIFTP  is  currently  being  implement¬ 
ed  by  UCL  as  described  in  Chapter  4.  Several  candidate 
terminal  protocols  are  being  considered  by  the  working  groups, 
and  UCL  has  done  a  test  implementation  of  one  of  these. 

Work  on  a  transport  station  interface  is  an  urgent  matter 
for  both  ISO  and  CCITT;  in  order  to  avoid  the  need  to  add 
a  protocol  such  as  TCP,  the  transport  station  being  designed, 
assumes  that  lower  layers  can  both  reliably  deliver  messages 
and  also  cope  with  flow  control  problems  that  may  arise. 

Within  the  UK  a  candidate  protocol  has  been  developed  with 
an  INDRA  member  on  the  working  parties  concerned,  and  UCL 
has  promised  to  implement  and  evaluate  that  protocol,  when 
it  has  been  defined. 

The  INDRA  Group  members  are  involved  in  the  following  Network 
and  Standards  Committees: 

Higginson  on  PO  Study  Group  2  -  Higher  level  Protocols  and 
PO  Study  Group  3  -  Transport  Services 

Higginson  and  Kirstein  attend  different  subgroups  on 

PO  Study  Group  4  -  PSS  Transition 

FTP  Implementors  Group 

Bennett,  Fisher,  Frost,  Higginson. 

UK  Euronet  AD  HOC  Host  Group 

Kirstein  is  Chairman,  Higginson  attends. 

ISO  TC/SC16  (Open  System  Connection) 

Higginson  is  rapporteur  of  WG  1  amd  WG  2. 
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BSI  DPS  20  (UK  counterpart  of  ISO  Committee  SC16) 

Hi^Kinson  is  on  the  Committee 

Hinchley  is  on  the  Model  Group  WG  1 

Bradbury  is  on  the  Task  Activation  Group  WG  2 

Hinchley  is  on  the  Transport  Station  Group  WG  3 

Higginson  is  Chairman  of  the  Terminal  Protocol  Group  WG  4 

In  addition  there  are  many  other  specific  network  groups  of 
the  University  of  London,  and  the  Inter-University  Computer 
Committee. 

Related  to  these  activities,  two  important  documents  were 
produced.  One  (13)  investigated  in  depth  possible  designs 
for  bulk  transfer  systems  for  Euronet.  It  then  defined 
a  Remote  Printing  Protocol,  and  considers  how  the  protocol 
might  be  implemented.  This  work  is  being  published  by  the 
Commission  of  the  EEC  for  Euronet  use.  The  second  document 
(35)  is  the  definitive  account  of  a  2  week  ISO  TC  97/SC16 
meeting  to  define  the  Presentation  Layer  of  the  proposed 
"ISO  layered  Model  of  Protocols  for  Open  Systems 
Interconnection" . 
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X.  COLLABORATORS 

The  following  people'  collaborated  in  the  work  of  this 
report  during  1978.  A  indicates  that  the  appointment 
was  made,  an  that  it  ceased,  during  1978;  Individuals 
carrying  both  indications  changed  their  status  during  the 
year. 

Academic  Staff  :  P.L.  Higglnson,  P.T.  Kirstein,  S.R.  Wilbur 
Technical  Supervisor  ;  A.R.  Duncan*,  H.R.  Gamble^ 

Research  Fellow  ;  A.J.  Hinchley 

Research  Assistants  ;  C.J.  Bennett^  C.  Bradbury,  R.C.  Cringle, 
S.  Das,  S.W.  Treadwell,  S.  Yilmaz. 

Engineer  :  B.  Jones*,  H.R.  Gamble*. 

Research  Student  :  A.  Akinpelu,  C.J.  Bennett*,  S.W.  Edge, 

Z.Z.  Fisher,  J.O.  Johnson  ,  P.  Lloyd  . 

I# 

Visitor  :  E.  Getz 

Secretaries;  M.  Massey,  E.O.  Oakley. 
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XI.  PUBLICATIONS 

We  list  below  the  publications  written  during  1978.  We  include 
also  some  publications,  marked  with  *  ,  which  were  mentioned 
in  the  previous  report  but  without  page  numbers  because  they 
were  "in  the  press". 

1.  Bradbury,  C.  X25  Asymmetries  and  how  to  Avoid  them.  Comp. 
Comm.  Rev.  8,  3,  25-34,  1978. 

2.  Bennett,  C.J.  Supporting  Transnet  Bulk  Data  Transfer. 

Proc .  Symp.  Flow  Control  in  Computer  Networks,  Paris, 

383-404,  1979. 

3.  Cerf,  V.G.  and  P.T.  Kirstein.  Issues  of  Packet  Network 
Interconnection.  Proc.  IEEE  66,  pp.  1386-1408,  1978. 

4.  Edge,  S.W.  Comparison  of  the  Hop-by-Hop  and  Endpoint 
Approaches  to  Network  Interconnections.  Proc.  Symp.  Flow 
Control  in  Computer  Networks,  Paris,  359-377. 

5.  Edge,  S.W.  and  A.J.  Hinchley.  A  Survey  of  End-to-End 
Retransmission  Techniques.  Comp.  Comm.  Rev.  8,4,  1-18,  1978. 

6.  Grossman,  G.R.  and  A.J.  Hinchley.  Issues  in  International 
Public  Data  Networks.  Proc.  USA-Japan  Conference,  1979. 

7.  Higginson,  P.L.  The  Design  of  Bulk  Data  Transfer  Systems 
in  Heterogeneous  Computer  Networks.  Proc.  SEAS  78,  Stresa, 
1978. 

8.  Higginson,  P.L.  The  Design  of  Bulk  Transfer  Systems  for 
Euronet .  To  be  published  by  the  Commission  of  the  European 
Community,  Luxembourg,  1979. 

9.  Hinchley,  A.J.  Some  Service  Aspects  of  the  X25  Interface. 
Advanced  Study  Institute  on  Interlinking  Computer  N«jtworks. 
North  Holland  1979. 
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10.  Hinchley,  A.J.  and  C.J.  Bennett.  Gateways  to  Computer 

Networks.  Proc .  Networkshop  2,  Liverpool  U.  105-121, 

1978. 

11.  Kent,  S.A.  A  National  Facsimile  and  Text  Virtual  System, 
Small  System  Software,  3,  2-13,  1978. 

12.  Kirstein,  P.T.  Choice  of  Data  Communications  Media  for 
Transmission  of  Facsimile  Information.  Computer  Networks, 
2,  179-190,  1978. 

13.  Kirstein,  P.T.  Some  International  Developments  in  Data 
Services.  ASI  on  Interlinking  Computer  Networks,  North 
Holland,  1979. 

14.  Kirstein,  P.T.  and  S.  Yilmaz.  Facsimile  Transmission  in 
Message  Processing  Systems  with  Data  Management.  ICCC, 
1978,  Kyoto,  717-725,  1978. 

15.  Sunshine,  C.  and  A.J.  Hinchley.  Submission  (by  IFIP)  to 
CCITT.  Implications  of  recommendation  X25  and  proposed 
improvements  for  Public  Data  Network  Interconnection. 
CCITT,  1978. 
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